acceptodds
Under review as a conference paper at ICLR 2027

ReplayWorld: Evolving Agentic World Model for Agent Training

Abstract

LLM-based environment simulators increasingly synthesize trajectories for agent training, reducing reliance on costly real environments. However, prompted and fine-tuned simulators generalize poorly across environments, and their environment knowledge stays fixed after deployment. We introduce ReplayWorld, an agentic world model for eight environments in web navigation, CoWork and tool use. It searches for information, executes code and consults a repertoire of simulator skills that encode how environments respond to actions. Replay-driven skill evolution writes recurring discrepancies between predicted and recorded observations into these skills, so environment knowledge evolves without weight updates. Semantic balanced sampling (SBS) promotes replay diversity, learnability-prioritized filtering (LPF) focuses selection on learnable traces, and multi-criteria cycle arbitration (MCCA) selects the best repertoire through independent validation. A hierarchical skill router (HiSkill) routes relevant skills during trajectory synthesis. On ReplayBench, our benchmark of 2,080 real environment transitions scored on six criteria, ReplayWorld outperforms its prompted backbone by 4.2 points. The agent trained on ReplayWorld trajectories gains 12.0 points on average across eight agent benchmarks. Code is available at https://anonymous.4open.science/r/replayworld-anon-2027-3ABB/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, LLM agents, agentic world model, skill evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UkF2OpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ReplayWorld builds an environment simulator by converting discrepancies between simulated and recorded observations into editable skills, without updating model weights. Its replay loop selects trajectories and revises skills. The paper introduces ReplayBench across eight environments and evaluates downstream agent training. Results show improvements over the prompted backbone.

Critical strengths

Explicit skill updates offer a practical way to adapt simulators without retraining model weights, which is more adaptive and applicable to different scenarios.

Critical weaknesses
  1. Trajectory separation does not ensure task separation. Four benchmarks contribute additional runs of evaluation tasks to the replay corpus, which causes the benchmark score to suffer from overfitting.
  2. It is weird that GPT -6 Astra is worse than Opus 5, and some subdomains are worse than GPT-5.6 Sol, which makes the benchmark score not quite convincing.
  3. Simple retrieval already scores 58.6 versus ReplayWorld's 59.4, which compromises the utility of the proposed ReplayWorld.
Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer UkF2 (1/4)

We sincerely thank you for the careful and constructive review and your thoughtful feedback on task generalization, benchmark reliability, and the value of skill evolution. We address each point below with clarifications and additional results.


[W1-1. Task-level Generalization & Task-specific Memorization] Trajectory separation does not ensure task separation; additional runs of evaluation tasks in the corpus could lead to overfitting.

Thank you for this thoughtful question. We have further clarified task-query construction in Appendix C.1 (Table S1) and added a similarity-filtered evaluation (Table S2).

  • [A. Task Difference] ReplayBench has no identical task-query matches with the skill-evolution training corpus and shares no trajectory with it. The training corpus, downstream SFT query pool, and ReplayBench are constructed and selected separately. Shared environments, tools, and task types allow us to study whether learned environment knowledge transfers to new queries within the same environment.
  • [B. Construction across Environments] Table S1 details the sources and collection for all eight environments. OS uses different generated instructions for training and evaluation. ACEBench includes new questions based on official tools and class definitions, with some variants also changing the initial state. SheetCopilot uses generated training queries and official evaluation queries absent from the training corpus. Downstream SFT uses its own generated query pool; the selected skills contain no evaluation-task text.
  • [C. Task-Query Similarity Analysis] Inspired by GraphWalker [1], we analyze similarity beyond exact matches using BGE-M3 embeddings and a cosine-similarity threshold of 0.85. Each evaluation query is compared with training and reference queries from the same environment, using query text without shared tool descriptions. Queries with maximum similarity ≥ 0.85 are removed together with their traces. Both simulators use the same retained subset (Appendix C.1, Table S2).
Evaluation subset# TracesPrompted 397BReplayWorld-397BΔ
Full ReplayBench2,08055.259.4+4.2
With a strict similarity filter (τ = 0.85)2,01354.959.2+4.3

ReplayWorld retains a +4.3-point advantage after highly similar queries are removed, compared with +4.2 on the full benchmark. This supports the robustness of our agentic world model to training–evaluation query similarity. The improvement persists when evaluation queries have no match at or above the specified similarity threshold.

Connection to Skill Evolution. Skill evolution samples from a large corpus at each round without exhaustively replaying it up to the selected round, so similar queries present in the corpus need not have contributed to the selected skills. The similarity-filtered evaluation directly checks the stability of this advantage.

[1] Xu et al. GraphWalker: Agentic Knowledge Graph Question Answering via Synthetic Trajectory Curriculum. 2026, Appendix B.2.


[W1-2. Generalization beyond ReplayBench] Could the gains reflect task-specific knowledge rather than transferable environment knowledge?

We would like to respectfully clarify that we have already provided OOD evaluation on another benchmark, AgentWorldBench, in the main paper (Sec. 4.2, Table 2) and appendix (App. E.1, Table 13).

  • Different domains. AgentWorldBench covers MCP, Search, Terminal, SWE, Android, Web, and OS, including SWE and Android beyond ReplayBench’s domain coverage.
  • Different tasks and benchmark sources. Its sources include Tool Decathlon, Terminal-Bench 1.0/2.0, AndroidWorld, and OSWorld-Verified. These differ from ReplayBench’s eight environments. Tasks include code editing and test execution, Android calendar and file-management operations.

We have compared ReplayWorld with LLM-as-WM (Table 2; App. E.1, Table 13); selected results are shown below:

MetricPrompted 397BReplayWorld-397BΔ
Average64.367.4+3.1
Consistency72.681.3+8.7
LongHorizon32.939.4+6.5

The benefits of evolved skills extend beyond ReplayBench, including improved state consistency and multi-step simulation on other benchmarks and domains. Together with the query-similarity analysis, these results support the value of reusable environment knowledge across different evaluation tasks.

AuthorsOpenReview ↗

Response to Reviewer UkF2 (2/4)

[W2-1. Model Rankings on ReplayBench] GPT-6-astra scores below Opus 5 and below GPT-5.6-sol on some subdomains, raising concerns about the benchmark.

Thank you for this careful observation. We would like to respectfully clarify the comparisons in Table 1. GPT-6-astra is among the two highest-scoring models: 63.6 Overall, ahead of GPT-5.6-sol (61.6) and only 0.1 below Opus 5 (63.7). It also leads on CoWork Judge (71.4), CoWork Rule (55.6), and Tool Use Rule (62.3). Table 12 reports the individual criteria (App. E.1):

CriterionClaude Opus 5GPT-6-astraGPT-5.6-sol
Format87.388.989.8
EnvReaction54.257.848.9
Consistency85.288.188.0
Factuality56.850.649.7
LongHorizon45.943.742.8
ActionPreserve52.852.550.3
Overall63.763.661.6
  • [A. Overall and Criterion-level Performance] GPT-6-astra leads GPT-5.6-sol on five of six criteria; its only lower score is Format (88.9 vs. 89.8), a gap of just 0.9 points. GPT-5.6-sol is slightly ahead on Web Navigation Judge (61.3 vs. 59.8) and Tool Use Judge (70.8 vs. 70.3), while GPT-6-astra leads substantially on Tool Use Rule (62.3 vs. 50.2; Table 1).
  • [B. Response Behavior vs. Factual and Multi-step Accuracy] GPT-6-astra leads Opus 5 on EnvReaction (57.8 vs. 54.2), Consistency (88.1 vs. 85.2), and Format. Opus 5 leads on Factuality (56.8 vs. 50.6), LongHorizon (45.9 vs. 43.7), and ActionPreserve. Thus, the models differ in predicting response types and maintaining state consistency versus preserving factual content over a rollout, despite nearly equal Overall scores.
  • [C. CoWork vs. Web Navigation and Tool Use] GPT-6-astra leads Opus 5 on CoWork Judge (71.4 vs. 68.1) and Rule (55.6 vs. 53.0), while Opus 5 leads on Web Navigation Judge (62.3 vs. 59.8) and Tool Use Judge (73.5 vs. 70.3; Table 1). CoWork involves files, spreadsheets, and applications; performance there need not track web or tool-response accuracy.

Within model families, Opus 5 exceeds Opus 4.6 (63.7 vs. 60.1), and Qwen3-14B exceeds Qwen3-8B (51.5 vs. 49.2; Table 1). ReplayBench captures broad capability improvements while distinguishing models’ strengths in environment simulation. The cross-judge and human validation below further support ReplayBench’s reliability (Sec. 4.2; Apps. D.3–D.4, Tables 8 and 10).


[W2-2. Cross-judge, Human, and Evaluation-agent Validation] How convincing are the benchmark scores and model rankings?

Thank you for your invaluable feedback. We have already provided cross-judge and human validation in Sec. 4.2 and Appendices D.3–D.4. We also add a comparison using a different ActionPreserve agent.

[A. Judges from Different Model Families] Table 10 reports the mean of Format, Consistency, and Factuality under three judges:

SimulatorGPT-5.6-terra judgeDeepSeek-V3.2 judgeClaude Sonnet 5 judge
Claude Opus 576.463.764.3
GPT-6-astra75.862.762.9
GPT-5.6-sol75.960.760.8
Prompted 397B71.756.354.7
ReplayWorld-397B74.259.658.9

Opus 5 scores above GPT-6-astra under all three judges, including the GPT judge. Pairwise ranking correlations in the validation study are 0.966, 0.988, and 0.965 (Table 8; App. D.3). ReplayWorld-397B also improves over Prompted 397B under every judge. The benefit of evolved skills is therefore consistent across the three judging models.

[B. Agreement with Human Annotations] Three annotators independently evaluate 100 traces. Factuality agrees with their mean at Spearman 0.80, and response-type classification agrees with human labels at κ = 0.911 (App. D.4). These checks validate both predicted content and whether the simulator returns the appropriate response type.

[C. Further Analysis for ActionPreserve Metric] ActionPreserve measures whether an agent takes the same next action after a simulated observation as after the real one. Using the same agent for each method provides a controlled comparison. We further replace Qwen3.5-397B with GLM-5:

SimulatorActionPreserve, Qwen3.5-397B agentActionPreserve, GLM-5 agent
Claude Opus 552.851.6
GPT-6-astra52.553.1
DeepSeek-V3.246.745.8
Qwen3.5-397B-A17B46.245.6
ReplayWorld-397B47.547.9

The overall performance pattern remains similar across evaluation agents, with no clear preference for any particular model family. Together, cross-judge validation, human annotation agreement, and cross-family checks support the reliability of ReplayBench’s main comparisons.

Thank you again for your thoughtful feedback and careful reading of our work!

AuthorsOpenReview ↗

Response to Reviewer UkF2 (3/4)

[W3-1. Plain Retrieval (Ref.) vs. Skills] Retrieval performance relative to ReplayWorld raises questions about the value of skill evolution.

Thank you for this thoughtful question about what skill evolution adds beyond retrieval to certain reference data. We provide controlled comparisons that make the contribution of Skills and Ref. explicit.

[A. What Ref. and Skills Provide] With Ref., the harness provides a reference file containing a few-shot sample (3 logged records from the training corpus for each environment) (Sec. 3.3; Table 4; App. E.4). Our aim is to study how different forms of access to training experience affect our agentic world model: Ref. provides concrete examples, whereas Skills provides compact environment knowledge distilled and refined through iterative replay. The skills evolve from the training corpus, which shares no trajectory with ReplayBench, and contain no evaluation-task text (Sec. 4.1; App. C.1, Eq. 10). Skill evolution extracts reusable rules and revises them in response to prediction errors, helping the simulator apply experience beyond individual examples.

[B. Controlled Ref./Skills Comparisons] Combined with Table 4, we provide comparisons across OpenCode, Claude Code, and Codex. Within each column, the backbone, harness, and generation-token limit are fixed; within each four-row group, Search + Coding is fixed. Gains in ( ) are relative to the corresponding baseline without Skills or Ref. (App. E.4, Table S3):

SkillsRef.Search + CodingOpenCodeClaude CodeCodex
✗✗✗54.055.455.9
✗✓✗55.1 (+1.1)56.3 (+0.9)57.0 (+1.1)
✓✗✗56.8 (+2.8)58.2 (+2.8)59.0 (+3.1)
✓✓✗57.058.459.1
✗✗✓55.657.157.8
✗✓✓56.9 (+1.3)58.4 (+1.3)59.2 (+1.4)
✓✗✓58.2 (+2.6)59.5 (+2.4)60.3 (+2.5)
✓✓✓59.461.061.7

[C. What Skills Add] Adding Skills alone improves performance by 2.4–3.1 points, compared with 0.9–1.4 points from Ref. alone. Skills remain useful when reference examples are already available: adding Skills to Ref. yields a further 1.9–2.6 points across the six harness/tool settings. Their combination performs best under all three harnesses.

Evolving skills through iterative refinement provides substantial and useful environment knowledge, while reference examples provide complementary information. Skills also help with Search + Coding enabled: learned environment knowledge and agentic capabilities contribute together to an effective agentic world model.


[W3-2. Skill Transfer and Adaptation] What useful capability does skill evolution provide beyond supplying logged examples?

We provide further analysis of how learned environment knowledge can be reused across different models and updated as environments change.

[A. Reuse across Different Models] Our evolved skill repertoire can be reused by other models as agentic world models. The table compares different models with and without our evolved skill repertoire:

ModelBackbone+ Our Evolved Repertoire
Qwen3.5-397B-A17B55.259.4
Claude Sonnet 560.564.8
Claude Opus 563.767.3

Evolved skills also benefit stronger models, including Sonnet 5 and Opus 5. The model that evolves the knowledge need not be the model that uses it. This indicates that ReplayWorld is a scalable approach.

[B. Updating Knowledge when the Environment Changes] After the OS changes from Ubuntu to Alpine, one replay cycle on 10 new traces raises the outcome-agreement advantage over the no-repertoire baseline from 4.5 to 15.0 points (Sec. 4.5; App. E.6, Table 15). Skill evolution supports both reuse and revision of environment knowledge: the simulator can retain useful rules and refine them when new behavior is observed.

Reviewer_cAzYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReplayWorld represents environment knowledge as editable simulator skills and updates them by comparing predicted observations with recorded transitions. Semantic balanced sampling, learnability-prioritized filtering, and validation-based repertoire selection support skill evolution. The paper also introduces ReplayBench and evaluates downstream agents trained on simulated trajectories across eight environments, including adaptation to an operating-system change.

Critical strengths
  1. Updating explicit simulator knowledge without changing model weights is a practical approach to maintaining environment simulators. The Ubuntu-to-Alpine experiment provides concrete evidence that replay can repair outdated environment assumptions.

  2. The evaluation connects observation prediction with downstream agent performance across diverse environments. ReplayBench includes complementary criteria, multiple-judge checks, and human validation, making the benchmark a potentially useful contribution beyond the proposed simulator.

Critical weaknesses
  1. Trajectory-level separation does not consistently establish task-level generalization. Appendix C.1 states that four environments contribute other executions of evaluation tasks to the replay corpus. This does not imply direct copying of evaluation trajectories, but the repertoire can still acquire task-specific knowledge. Consequently, results on these environments cannot cleanly establish generalization to unseen tasks.

  2. Several comparisons do not isolate simulator quality. Appendix B documents differences in query pools, filtering, and some policy-training settings; observation-prediction budgets also differ across configurations. Moreover, retrieval-grounded prompting reaches 58.6 Overall versus ReplayWorld’s 59.4 (Appendix E.4), substantially narrowing the improvement over a strong experience-based alternative. The downstream gains over prompted-simulator training are also smaller than the headline gains over untrained agents. These issues make the incremental value of skill evolution difficult to quantify.

Questions for authors
  1. Can you provide a task-disjoint evaluation for the environments whose replay corpora contain alternative executions of evaluation tasks?

  2. Under matched query pools, filtering, training-token budgets, and inference capabilities, how much improvement remains over prompted and retrieval-grounded simulators?

7 comments
AuthorsOpenReview ↗

Response to Reviewer cAzY (1/4)

We appreciate the time you spent reviewing our paper and your insightful feedback! We address each point below with clarifications and additional results.


[W1-1 & Q1. Task-level Generalization & Task-specific Memorization] Trajectory-level separation does not consistently establish task-level generalization; skills may acquire task-specific knowledge. Can you provide a task-disjoint evaluation where the corpus contains alternative executions of evaluation tasks?

Thank you for this careful question about task-level generalization. We have further clarified task-query construction and collection in Appendix C.1 (Table S1) and added a similarity-filtered evaluation (Table S2).

  • [A. Task Difference] ReplayBench has no identical task-query matches with the skill-evolution training corpus and shares no trajectory with it. The skill-evolution corpus, downstream SFT query pool, and ReplayBench are constructed and selected separately. They can share environments, tools, and task types, allowing us to study whether learned environment knowledge transfers to new queries within the same environment. Table S1 details construction and collection for each environment; downstream SFT uses its own generated query pool.
  • [B. Task-Query Similarity Analysis] Inspired by GraphWalker [1], we analyze task-query similarity beyond exact matches and apply a strict BGE-M3 filter with a cosine-similarity threshold of 0.85. Each evaluation query is compared with training and reference queries from the same environment, using task-query text without shared tool descriptions. Queries with maximum similarity ≥ 0.85 are removed together with their associated traces. Both simulators are evaluated on the same retained subset (Appendix C.1, Table S2).
Evaluation subset# TracesPrompted 397BReplayWorld-397BΔ
Full ReplayBench2,08055.259.4+4.2
With a strict similarity filter (τ = 0.85)2,01354.959.2+4.3

ReplayWorld retains a +4.3-point advantage after highly similar queries are removed, compared with +4.2 on the full benchmark. This supports the robustness of our agentic world model to training–evaluation query similarity. The improvement persists when evaluation queries have no match at or above the specified threshold.

Connection to skill evolution. The replay procedure samples from a large corpus at each round without exhaustively replaying it up to the selected round. A similar query in the corpus therefore need not have been used to evolve the selected skills. The similarity-filtered evaluation directly tests the stability of the observed advantage.

[1] Xu et al. GraphWalker: Agentic Knowledge Graph Question Answering via Synthetic Trajectory Curriculum. 2026, Appendix B.2.


[W1-2. Generalization to Unseen Tasks] Do the results establish generalization to unseen tasks?

Thank you for emphasizing the importance of generalization. We would like to respectfully clarify that we have already provided OOD evaluation on other benchmarks (AgentWorldBench) in the main paper (Sec. 4.2, Table 2) and appendix (App. E.1, Table 13). This independently constructed benchmark provides evidence beyond ReplayBench and the query-similarity check.

  • Different domains. AgentWorldBench covers MCP, Search, Terminal, SWE, Android, Web, and OS, including SWE and Android, which are outside ReplayBench’s domain coverage.
  • Different tasks and benchmark sources. Its nine sources are MCPMark, Tool Decathlon, WideSearch, Terminal-Bench 1.0 and 2.0, an in-house SWE benchmark, WebArena-Verified, AndroidWorld, and OSWorld-Verified. These differ from ReplayBench’s eight environments. Concrete tasks include code editing and test execution, Android calendar and file-management operations.

We have compared ReplayWorld with LLM-as-WM (Table 2; App. E.1, Table 13); selected results are shown below:

MetricPrompted 397BReplayWorld-397BΔ
Average64.367.4+3.1
Consistency72.681.3+8.7
LongHorizon32.939.4+6.5
  • State consistency: the +8.7-point improvement measures whether the simulator preserves entities, values, and state from the interaction history.
  • Multi-step simulation: the +6.5-point LongHorizon improvement measures Consistency and Factuality when the simulator continues from its own predicted observations.

These results show that the benefits of evolved skills extend beyond ReplayBench, including improving state consistency and multi-step simulation on other benchmarks and domains.

AuthorsOpenReview ↗

Response to Reviewer cAzY (2/4)

[W2-1 & Q2. Prompted Simulator and ReplayWorld: Settings] Differences in query pools, filtering, training budgets, and inference capabilities; A controlled comparison of simulator quality?

Thank you for your careful question. We have further clarified the shared training and evaluation settings in Appendix B.3 and added the training-pool counts in Table S4.

  • [A. Training and Evaluation] Agents trained with the prompted simulator (LLM-as-WM) and our agentic world model, ReplayWorld, use the same SFT recipe, training epochs, hyperparameters, training-token budget, test configuration, and scoring protocol within each benchmark and policy size. Each benchmark has its own metric, task set, and context length; settings can differ across benchmarks (Sec. 4.1; App. B.3).
  • [B. Queries and Trajectory Counts] Both simulators use the same task queries and the same number of training trajectories. Table S4 reports the counts per simulator before step-level SFT expansion:
Benchmark# Training Traj.
WebShop6,996
Mind2Web1,199
OS1,000
SheetCopilot2,207
OfficeBench6,666
ACEBench6,101
BFCL3,832
τ²-bench3,200

For τ²-bench, 800 shared task queries yield 3,200 trajectories per simulator, subsequently expanded into step-level SFT examples. Under these shared settings, ReplayWorld improves all three domain averages at both policy sizes in Table 3.

  • [C. Inference Capabilities and Budgets] Our additional Ref./Skills comparisons use the same backbone, harness, and generation-token limit within each column; Search + Coding is fixed within each group. The comparison varies access to Skills and Ref. (see also [W2-2 & Q2] below).

[W2-2 & Q2. Plain Retrieval (Ref.) vs. Skills] Retrieval-grounded prompting v.s. ReplayWorld; how much improvement remains under matched settings?

Thank you for this thoughtful question about what skill evolution adds beyond retrieval to certain reference data.

[A. What Ref. and Skills Provide] With Ref., the harness provides a reference file containing a few-shot sample (3 logged records from the training corpus for each environment) (Sec. 3.3; Table 4; App. E.4). Our aim is to study how different forms of access to training experience affect our agentic world model: Ref. provides concrete examples, whereas Skills provides compact environment knowledge distilled and refined through iterative replay. The skills are evolved from the training corpus, which shares no trajectory with ReplayBench, and no selected skill contains evaluation-task text (Sec. 4.1; App. C.1, Eq. 10). Skill evolution extracts reusable rules from experience and revises them in response to prediction errors, allowing the simulator to apply what it learns beyond individual examples.

[B. Controlled Ref./Skills Comparisons] Combined with Table 4, we provide comprehensive comparisons across OpenCode, Claude Code, and Codex. Within each column the harness and generation-token limit are fixed; within each four-row group Search + Coding is fixed. Gains in ( ) are relative to the corresponding baseline without Skills or Ref. (App. E.4, Table S3):

SkillsRef.Search + CodingOpenCodeClaude CodeCodex
✗✗✗54.055.455.9
✗✓✗55.1 (+1.1)56.3 (+0.9)57.0 (+1.1)
✓✗✗56.8 (+2.8)58.2 (+2.8)59.0 (+3.1)
✓✓✗57.058.459.1
✗✗✓55.657.157.8
✗✓✓56.9 (+1.3)58.4 (+1.3)59.2 (+1.4)
✓✗✓58.2 (+2.6)59.5 (+2.4)60.3 (+2.5)
✓✓✓59.461.061.7

[C. What the Comparisons Show] Adding Skills alone improves performance by 2.4–3.1 points, compared with 0.9–1.4 points from Ref. alone. Skills also improve performance when reference examples are already available: adding Skills to Ref. yields a further 1.9–2.6 points across the six harness/tool settings. The combination performs best under all three harnesses.

Evolving skills through iterative refinement already provides substantial and useful environment knowledge, while concrete reference examples provide complementary information. Skills remain useful with Search + Coding enabled: tool access and learned environment knowledge contribute together to an effective agentic world model.

AuthorsOpenReview ↗

Response to Reviewer cAzY (3/4)

[W2-3 & Q2. ReplayWorld vs. Prompted Simulator for Downstream Agent Training] The downstream gains over prompted-simulator training are smaller than the gains over untrained agents. What is the incremental benefit of ReplayWorld?

Thank you for this thoughtful question about the practical value of our agentic world model. Agents trained with ReplayWorld outperform those trained with the prompted simulator (LLM-as-WM) across all three domain averages at different scales in Table 3. We further compare downstream agent performance (overall score) across seeds and larger policy sizes.

[A. Seeds and Larger Policies] Our default seed is 42, as specified in the anonymous code repository: replayworld/config.py (ClusterConfig.seed, EvolveConfig.seed). Agent fine-tuning also uses seed 42 for the results in Table 3. We add runs with seeds 43 and 44 and report mean ± SD across the three seeds. For the multi-run trajectory-synthesis experiments, we regenerate the SFT data separately for each run, so each agent is fine-tuned on a different generated dataset. We also extend agent training to Qwen3-32B:

PolicySettingLLM-as-WMReplayWorld
Qwen3-8BSeed 42 (Table 3)43.946.0
Qwen3-8BMulti-run for agent training43.53 ± 1.2246.07 ± 0.54
Qwen3-8BMulti-run for trajectory synthesis43.46 ± 1.3646.79 ± 0.69
Qwen3-14BSeed 42 (Table 3)46.850.3
Qwen3-14BMulti-run for agent training47.20 ± 0.4850.63 ± 0.29
Qwen3-32BMulti-run for agent training51.02 ± 0.7854.33 ± 0.27

ReplayWorld is scalable and maintains its advantage over prompted-simulator training at larger policy sizes. The separate experiments also show that the improvement is not limited to the training pool generated in a single run.

[B. A Smaller Simulator Backbone] The result also holds when the simulator uses Qwen3.6-27B as the backbone for our agentic world model. We compare agents trained with LLM-as-WM-27B and ReplayWorld-27B:

EvaluationQwen3-8B: LLM-as-WM-27BQwen3-8B: ReplayWorld-27BΔQwen3-14B: LLM-as-WM-27BQwen3-14B: ReplayWorld-27BΔ
Overall41.844.9+3.146.750.1+3.4

ReplayWorld-27B improves downstream performance at different agent scales, with gains of 3.1 and 3.4 points. These comparisons measure the practical effect of simulator-generated data on agents evaluated in real environments.

[C. Practical Insight] As an agentic world model, ReplayWorld serves as a useful simulator for agent training: improved performance across various domains shows that better environment simulation can support more effective agent behavior in real environments (Table 3). Together with the skill-transfer results below, these findings highlight the scalability of ReplayWorld. Evolved skills improve environment simulation across backbones and also benefit larger downstream policies, allowing our agentic world model to serve as a capable simulator for agent training across domains.

Reviewer_3ZsnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReplayWorld, an LLM-based environment simulator for generating agent training data that keeps its knowledge of each environment as editable text "skills" instead of in model weights. The skills are learned by replaying logged trajectories and turning recurring prediction errors into edits, and the simulator can also run code and search the web while predicting. The authors introduce ReplayBench, a validated benchmark for next-observation prediction, and show that ReplayWorld beats its prompted base model there, that agents trained on its simulated trajectories improve on real benchmarks, and that the skills can be updated cheaply when an environment changes.

Critical strengths
  1. ReplayBench is a solid, carefully checked benchmark and probably the paper's most lasting contribution. It covers eight environments and separately scores format, state tracking, factual content, response type, multi-step rollouts and whether an agent would act the same way. The validation is quite careful: three judges from different model families rank systems almost identically, the factuality judge agrees with human annotators about as well as they agree with each other, and the response-type classifier is checked against humans too. The field doesn't really have a shared foundation for LLM world models yet, so this matters regardless of the method.

  2. Keeping environment knowledge as editable text skills is a sensible, practical design. The skills are readable and cheap to update, and the Ubuntu-to-Alpine experiment cleanly shows that one replay cycle on a handful of new traces adapts the simulator without retraining. Letting the simulator run code and search the web also handles cases a weights-only model can't.

Critical weaknesses
  1. What skill evolution itself adds is small, and it isn't compared against simpler ways of using the same data.
  • Plain retrieval (putting a few similar logged transitions from other tasks in the prompt) gets within about a point of the full system. But it is left out of the main table and the agent-training comparison. The stated reason, that retrieval needs the corpus to already hold the transition, applies to ReplayWorld too, because its skills and reference traces come from the same corpus.
  • The gain mostly goes away with the 27B backbone. In Figure 6 (four of its curves come from re-runs), the chosen skill set is no better than no skills on three of the eight environments.
  • There is no "tools but no skills" ablation, so we don't know what skills add once search and code are there. The SBS and LPF ablations use one environment and are smaller than the rerun noise the paper itself reports. HiSkill is never ablated.
  • Test tasks may overlap the training corpus. For four benchmarks, other runs of the same test tasks are in the replay corpus (the τ²-bench test traces cover 49 of its 50 airline tasks). Reference traces outside WebShop and Mind2Web aren't said to be task-disjoint. Only ReplayWorld can read this material, so part of the gain could be memory of specific tasks rather than general knowledge.
  1. The agent-training evidence is weaker than the headline says, and one headline number has no method behind it.
  • The "+12 points" is against the untrained model. The prompted simulator already gets most of that. ReplayWorld's own part is about two to three and a half points, from single runs without seeds.
  • The two simulators aren't trained the same way on some benchmarks, even though the paper says they are. SheetCopilot uses different queries and hyperparameters. The OfficeBench 14B baseline trains for only one epoch and barely moves off the untrained model. WebShop uses different filtering.
  • There is no comparison with trajectories from the real sandboxed environments, which is the natural upper bound. The AWM and EnvFactory rows in Table 3 are never explained. The RL result is one environment, one seed and a noisy curve.
  • The "hallucination rate on illegal actions" in the intro and Figures 1–2 isn't described anywhere, so there's no way to tell how it was measured.
  • Figure 1 compares ReplayWorld-27B with a weaker prompted model instead of its own backbone, so the gap looks much bigger than it is. The claim of beating prompted LLMs also skips the six proprietary prompted models that score higher.
Questions for authors
  1. How is the illegal-action hallucination rate measured? Which environments, how were the illegal actions made, how many were there, and how is a "normal observation" judged?

  2. What are the AWM and EnvFactory rows in Table 3? Did you train them on the same queries, or are they released checkpoints trained on other data?

  3. How many trajectories end up in each simulator's training pool? What happens on SheetCopilot and OfficeBench-14B with matched queries and hyperparameters?

  4. For the four fixed-task benchmarks, how many test traces have other runs of the same task in the corpus or the reference traces? How do the gains split between task-disjoint and overlapping traces?

  5. Does the 27B model learn its own skills or reuse the 397B ones? And since both the test traces and the ActionPreserve agent come from Qwen3.5-397B, does that favor simulators built on the same model?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 3Zsn (1/6)

We sincerely thank you for the careful and constructive review! We address each point below.


[W1-1. Plain Retrieval (Ref.) v.s. Skills] Plain retrieval performance & retrieval also needs the corpus.

Thank you for this thoughtful question about what skill evolution adds beyond retrieval to certain reference data. With Ref., the harness provides a reference file containing a few-shot sample (Sec. 3.3; Table 4; App. E.4) from training corpus (3 for each env). Our aim is to study how different forms of access to training experience affect our agentic world model: retrieval of Ref. provides few-shot concrete examples during training, whereas Skills provides compact environment knowledge distilled and refined through iterative replay. This tests the value of turning experiences into reusable simulator knowledge. Combined with Table 4, we provide comprehensive comparisons of Ref. and Skills across different harnesses (gains are over the baseline without Skills or Ref.):

SkillsRef.Search + CodingOpenCodeClaude CodeCodex
✗✗✗54.055.455.9
✗✓✗55.1 (+1.1)56.3 (+0.9)57.0 (+1.1)
✓✗✗56.8 (+2.8)58.2 (+2.8)59.0 (+3.1)
✓✓✗57.058.459.1
✗✗✓55.657.157.8
✗✓✓56.9 (+1.3)58.4 (+1.3)59.2 (+1.4)
✓✗✓58.2 (+2.6)59.5 (+2.4)60.3 (+2.5)
✓✓✓59.461.061.7

Adding Skills alone improves performance by 2.4–3.1 points, compared with 0.9–1.4 points from Ref. alone. Ref. also helps, and combining it with Skills further improves performance in all harnesses. These results indicate that evolving skills through iterative refinement already delivers substantial improvements, while concrete reference examples could provide complementary information.


[W1-2-1. 27B backbone performance] The gain mostly goes away with the 27B backbone.

Thank you for this thoughtful question. ReplayWorld-397B achieves a substantial gain of +4.2 Overall (55.2→59.4) on ReplayBench (Sec. 4.2; Table 1). ReplayWorld-27B also improves, although the gain is smaller than with the larger backbone (some further explanation is provided in [Q5-1]). Looking beyond the aggregate score, our 27B agentic world model shows clear gains in (1) prediction content, multi-step simulation, and the agent's subsequent actions (Table 12):

MetricPrompted 27BReplayWorld-27BΔ
Factuality44.747.9+3.2
LongHorizon46.348.0+1.7
ActionPreserve48.251.0+2.8

These improvements matter: Factuality tests the content of the predicted response; LongHorizon tests whether predictions remain useful when the simulator continues from its own outputs; ActionPreserve tests whether the simulated observation leads the agent to the same next action as the real observation. ReplayWorld-27B achieves the highest LongHorizon score in Table 12. Reconsulting environment skills at successive steps offers a way to retain environment rules as the simulated history grows, which is particularly relevant to free-running simulation;

(2) Domain-level results. The largest domain-level improvements are in CoWork (Table 1):

CoWork metricPrompted 27BReplayWorld-27BΔ
Judge64.067.2+3.2
Rule45.649.5+3.9

CoWork covers OS, SheetCopilot, and OfficeBench, where actions change files, spreadsheets, and application state. These gains are consistent with the value of explicit rules for simulating stateful environments.

(3) Usefulness for agent training. We additionally evaluate agents trained on trajectories generated by LLM-as-WM-27B and ReplayWorld-27B:

EvaluationQwen3-8B: LLM-as-WM-27BQwen3-8B: ReplayWorld-27BΔQwen3-14B: LLM-as-WM-27BQwen3-14B: ReplayWorld-27BΔ
Overall41.844.9+3.146.750.1+3.4

Training with ReplayWorld-27B improves downstream agentic task performance by 3.1 points for Qwen3-8B and 3.4 points for Qwen3-14B. This supports its usefulness as a env simulator for agent training at different scales.

AuthorsOpenReview ↗

Response to Reviewer 3Zsn (2/6)

[W1-2-2 Figure 6 ours vs w/o skills] In Figure 6, the chosen skill set is no better than no skills on certain environments.

Thank you for the careful reading. In Fig. 6, we aim to analyze how skill evolution improves our agentic world model and how MCCA helps select which round’s skill set to use. We would like to respectfully clarify that Fig. 6 reports the mean of five per-step criteria, excluding LongHorizon (App. E.2). LongHorizon tests whether the simulator maintains Consistency and Factuality over 3–12 steps using its own predicted observations. Generating and scoring these separate multi-step rollouts at all evolution rounds across 8 environments (figure6) would be computationally demanding, so we evaluate LongHorizon after evolution and include it in the final Overall score. After evolution, ReplayWorld improves on all six criteria over the prompted model (no skill) (Tables 1 and 12):

MetricPrompted 397B → ReplayWorld-397BΔ
Format89.8 → 90.1+0.3
EnvReaction29.7 → 40.4+10.7
Consistency79.8 → 80.4+0.6
Factuality45.5 → 51.9+6.4
LongHorizon40.3 → 46.1+5.8
ActionPreserve46.2 → 47.5+1.3
Overall55.2 → 59.4+4.2

After skill evolution, LongHorizon shows substantial gain, suggesting that evolved environment knowledge helps our agentic world model sustain coherent behavior across multiple steps. This improvement in long-horizon simulation ability increase during evolution, capturing a benefit that the per-step curves in Fig. 6 do not measure.

Insight from Fig.6: which round's skills to use. MCCA uses a separate arbitration score to select a round's skill set. Selecting this set instead of using the last round's skills improves both obs prediction quality and downstream agent performance (Table 5):

EvaluationLast round's skills → MCCA-selected skillsΔ
ReplayBench five-criterion mean58.4 → 60.7+2.3
Downstream Qwen3-8B on WebShop61.9 → 64.7+2.8

Skill evolution develops reusable environment knowledge; MCCA helps select an effective skill set for simulation and agent training.


[Q5-1. Skill source and transferability] Does the 27B model learn its own skills or reuse the 397B ones?

ReplayWorld-27B reuses the repertoire evolved by 397B. We additionally evaluate a repertoire evolved by the 27B and transfer the 397B-evolved repertoire to other backbones. Backbone denotes the prompted model.

ModelBackbone27B-evolved repertoire397B-evolved repertoire
Qwen3.6-27B56.957.157.5
Qwen3.5-397B-A17B55.2–59.4
Claude Sonnet 560.5–64.8
Claude Opus 563.7–67.3

Insightful findings. Regardless of which model evolves the skills, they encode reusable environment knowledge; benefiting from that knowledge also requires the simulator to interpret and apply it. A plausible explanation for the smaller 27B gain is that smaller backbones may struggle to faithfully execute the multiple conditions packed into compact skill instructions. This interpretation is consistent with FollowBench [1], which finds instruction-following performance declines as constraints accumulate, and RULER [2], which finds benefits from model scaling in long-context utilization. Our transfer results show that skill evolution remains useful across backbones, with larger improvements on ReplayWorld-397B, Sonnet 5, and Opus 5. The same evolved knowledge can therefore complement a stronger backbone, which may be better equipped to put those skills into practice.

[1] Jiang et al. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. ACL 2024.

[2] Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? COLM 2024.

AuthorsOpenReview ↗

Response to Reviewer 3Zsn (3/6)

[W1-3 Ablations: tools without skills, seed of SBS and LPF, HiSkill] “tools but no skills” ablation; SBS and LPF are tested on one environment; HiSkill should be ablated.

Thank you for the thoughtful suggestions. We consolidate and provide studies of harness capabilities, replay-loop seeds, and HiSkill selection as follows.

(a) Skills and harness capabilities. We compare Skills with Search, Coding, and Ref. access under three harnesses. ( ): improvement from adding Skills with the other capabilities fixed; < >: improvement from enabling tools with the Skills setting fixed.

SkillsSearchCodingRef.OpenCodeClaude CodeCodex
✗✗✗✗54.055.455.9
✓✗✗✗56.8 (+2.8)58.2 (+2.8)59.0 (+3.1)
✗✓✓✓56.9 <+2.9>58.4 <+3.0>59.2 <+3.3>
✓✓✓✓59.4 (+2.5) <+2.6>61.0 (+2.6) <+2.8>61.7 (+2.5) <+2.7>

Skills and harness capabilities both improve simulation, and their combination performs best. Search supplies external information, Coding enables execution, and Ref. provides concrete examples. In some harnesses, enabling these capabilities produces a larger improvement than adding Skills alone. This is consistent with our agentic design: the harness is an integral part of the agentic world model, and its capabilities complement the environment knowledge encoded in skills.

(b) SBS and LPF across seeds. Our original experiments use seed=42, specified by seed in replayworld/config.py in our anonymous code repository. We additionally run seeds 43 and 44 for the ablation studies in Table 5 on WebShop, OS, and ACEBench, covering each domain. Values are mean ± SD across the three seeds for the ReplayBench score.

VariantWebShopOSACEBench
Full60.71 ± 1.07561.99 ± 0.78757.03 ± 0.962
−MCCA58.38 ± 0.95461.78 ± 0.82656.27 ± 1.097
−MCCA −LPF58.12 ± 0.95160.91 ± 1.16055.69 ± 1.093
−MCCA −LPF −SBS57.39 ± 1.19759.40 ± 1.15054.55 ± 1.125

Removing all three modules reduces the score by 3.3, 2.6, and 2.5 points. These repeated runs support our main finding: useful skill evolution depends on both the coverage of replayed experience and the learnability of the errors selected for revision.

(c) HiSkill selection during trajectory synthesis (agentic world model as the simulator). [No skills] uses the prompted simulator (LLM-as-WM, Table 3); [Full index] lets the simulator browse the complete skill index; [Random] supplies randomly chosen operations; [HiSkill] selects relevant strategies and operations for the task (ReplayWorld, Table 3):

Synthesis Context (Skill Selection)WebShopOSBFCL
No skills61.627.158.4
Full index62.829.660.7
Random61.329.558.6
HiSkill64.732.361.4

Effective navigation through the skill repository helps the simulator locate and use the most relevant environment knowledge. Making that knowledge readily accessible helps realize the value of the learned skills in environment simulation and downstream agent training.

Comments on the paper

2 comments

Dear Reviewers,

Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.

Thank you!

AC

We sincerely thank all reviewers for your careful evaluation, thoughtful feedback, and follow-up discussion! We greatly appreciate the time and effort you have devoted to our work.

We summarize our responses below:

(W: weakness; Q: question.)

  1. We expanded ReplayWorld's downstream evaluations with repeated agent-training and trajectory-synthesis runs, larger policies, and the 27B simulator. (Reviewer 3Zsn [W2-1]; Reviewer cAzY [Q2]; Reviewer UkF2 [W3])

  2. We highlighted how evolved skills can be reused across different models and updated as environments change. (Reviewer 3Zsn [Q5]; Reviewer cAzY [W2]; Reviewer UkF2 [W3])

  3. We enlarged and consolidated the Ref./Skills comparisons with and without tool access across different harnesses. (Reviewer 3Zsn [W1-1]; Reviewer cAzY [Q2]; Reviewer UkF2 [W3])

  4. We supplemented our analysis with comparisons of HiSkill, full-index access, and random skill selection. (Reviewer 3Zsn [W1-3])

  5. We added real-environment and mixed-data training comparisons, alternative-simulator baselines, and multi-run RL results. (Reviewer 3Zsn [W2-3 & Q2])

  6. We explained how the illegal-action hallucination rate is measured. (Reviewer 3Zsn [W2-4 & Q1])

  7. We included query-similarity filtering to assess the robustness of ReplayWorld's performance. (Reviewer 3Zsn [Q4]; Reviewer cAzY [Q1]; Reviewer UkF2 [W1])

  8. We refined the supplementary writing on agent-training settings (B.3) and corpus sources and task construction (C.1), including proofreading corrections. We have further incorporated the valuable insights gained from the rebuttal discussion. (Reviewer 3Zsn [Q3; follow-up]; Reviewer cAzY [Q2])

We humbly hope this summary clarifies our responses. Thank you once again for your valuable time, thoughtful feedback, and consideration! 🙏

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.