- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReplayWorld builds an environment simulator by converting discrepancies between simulated and recorded observations into editable skills, without updating model weights. Its replay loop selects trajectories and revises skills. The paper introduces ReplayBench across eight environments and evaluates downstream agent training. Results show improvements over the prompted backbone.
Critical strengths
Explicit skill updates offer a practical way to adapt simulators without retraining model weights, which is more adaptive and applicable to different scenarios.
Critical weaknesses
- Trajectory separation does not ensure task separation. Four benchmarks contribute additional runs of evaluation tasks to the replay corpus, which causes the benchmark score to suffer from overfitting.
- It is weird that GPT -6 Astra is worse than Opus 5, and some subdomains are worse than GPT-5.6 Sol, which makes the benchmark score not quite convincing.
- Simple retrieval already scores 58.6 versus ReplayWorld's 59.4, which compromises the utility of the proposed ReplayWorld.
Questions for authors
See weakness.
6 comments
Response to Reviewer UkF2 (2/4)
[W2-1. Model Rankings on ReplayBench] GPT-6-astra scores below Opus 5 and below GPT-5.6-sol on some subdomains, raising concerns about the benchmark.
Thank you for this careful observation. We would like to respectfully clarify the comparisons in Table 1. GPT-6-astra is among the two highest-scoring models: 63.6 Overall, ahead of GPT-5.6-sol (61.6) and only 0.1 below Opus 5 (63.7). It also leads on CoWork Judge (71.4), CoWork Rule (55.6), and Tool Use Rule (62.3). Table 12 reports the individual criteria (App. E.1):
| Criterion | Claude Opus 5 | GPT-6-astra | GPT-5.6-sol |
|---|---|---|---|
| Format | 87.3 | 88.9 | 89.8 |
| EnvReaction | 54.2 | 57.8 | 48.9 |
| Consistency | 85.2 | 88.1 | 88.0 |
| Factuality | 56.8 | 50.6 | 49.7 |
| LongHorizon | 45.9 | 43.7 | 42.8 |
| ActionPreserve | 52.8 | 52.5 | 50.3 |
| Overall | 63.7 | 63.6 | 61.6 |
- [A. Overall and Criterion-level Performance] GPT-6-astra leads GPT-5.6-sol on five of six criteria; its only lower score is Format (88.9 vs. 89.8), a gap of just 0.9 points. GPT-5.6-sol is slightly ahead on Web Navigation Judge (61.3 vs. 59.8) and Tool Use Judge (70.8 vs. 70.3), while GPT-6-astra leads substantially on Tool Use Rule (62.3 vs. 50.2; Table 1).
- [B. Response Behavior vs. Factual and Multi-step Accuracy] GPT-6-astra leads Opus 5 on EnvReaction (57.8 vs. 54.2), Consistency (88.1 vs. 85.2), and Format. Opus 5 leads on Factuality (56.8 vs. 50.6), LongHorizon (45.9 vs. 43.7), and ActionPreserve. Thus, the models differ in predicting response types and maintaining state consistency versus preserving factual content over a rollout, despite nearly equal Overall scores.
- [C. CoWork vs. Web Navigation and Tool Use] GPT-6-astra leads Opus 5 on CoWork Judge (71.4 vs. 68.1) and Rule (55.6 vs. 53.0), while Opus 5 leads on Web Navigation Judge (62.3 vs. 59.8) and Tool Use Judge (73.5 vs. 70.3; Table 1). CoWork involves files, spreadsheets, and applications; performance there need not track web or tool-response accuracy.
Within model families, Opus 5 exceeds Opus 4.6 (63.7 vs. 60.1), and Qwen3-14B exceeds Qwen3-8B (51.5 vs. 49.2; Table 1). ReplayBench captures broad capability improvements while distinguishing models’ strengths in environment simulation. The cross-judge and human validation below further support ReplayBench’s reliability (Sec. 4.2; Apps. D.3–D.4, Tables 8 and 10).
[W2-2. Cross-judge, Human, and Evaluation-agent Validation] How convincing are the benchmark scores and model rankings?
Thank you for your invaluable feedback. We have already provided cross-judge and human validation in Sec. 4.2 and Appendices D.3–D.4. We also add a comparison using a different ActionPreserve agent.
[A. Judges from Different Model Families] Table 10 reports the mean of Format, Consistency, and Factuality under three judges:
| Simulator | GPT-5.6-terra judge | DeepSeek-V3.2 judge | Claude Sonnet 5 judge |
|---|---|---|---|
| Claude Opus 5 | 76.4 | 63.7 | 64.3 |
| GPT-6-astra | 75.8 | 62.7 | 62.9 |
| GPT-5.6-sol | 75.9 | 60.7 | 60.8 |
| Prompted 397B | 71.7 | 56.3 | 54.7 |
| ReplayWorld-397B | 74.2 | 59.6 | 58.9 |
Opus 5 scores above GPT-6-astra under all three judges, including the GPT judge. Pairwise ranking correlations in the validation study are 0.966, 0.988, and 0.965 (Table 8; App. D.3). ReplayWorld-397B also improves over Prompted 397B under every judge. The benefit of evolved skills is therefore consistent across the three judging models.
[B. Agreement with Human Annotations] Three annotators independently evaluate 100 traces. Factuality agrees with their mean at Spearman 0.80, and response-type classification agrees with human labels at κ = 0.911 (App. D.4). These checks validate both predicted content and whether the simulator returns the appropriate response type.
[C. Further Analysis for ActionPreserve Metric] ActionPreserve measures whether an agent takes the same next action after a simulated observation as after the real one. Using the same agent for each method provides a controlled comparison. We further replace Qwen3.5-397B with GLM-5:
| Simulator | ActionPreserve, Qwen3.5-397B agent | ActionPreserve, GLM-5 agent |
|---|---|---|
| Claude Opus 5 | 52.8 | 51.6 |
| GPT-6-astra | 52.5 | 53.1 |
| DeepSeek-V3.2 | 46.7 | 45.8 |
| Qwen3.5-397B-A17B | 46.2 | 45.6 |
| ReplayWorld-397B | 47.5 | 47.9 |
The overall performance pattern remains similar across evaluation agents, with no clear preference for any particular model family. Together, cross-judge validation, human annotation agreement, and cross-family checks support the reliability of ReplayBench’s main comparisons.
Thank you again for your thoughtful feedback and careful reading of our work!
Response to Reviewer UkF2 (3/4)
[W3-1. Plain Retrieval (Ref.) vs. Skills] Retrieval performance relative to ReplayWorld raises questions about the value of skill evolution.
Thank you for this thoughtful question about what skill evolution adds beyond retrieval to certain reference data. We provide controlled comparisons that make the contribution of Skills and Ref. explicit.
[A. What Ref. and Skills Provide] With Ref., the harness provides a reference file containing a few-shot sample (3 logged records from the training corpus for each environment) (Sec. 3.3; Table 4; App. E.4). Our aim is to study how different forms of access to training experience affect our agentic world model: Ref. provides concrete examples, whereas Skills provides compact environment knowledge distilled and refined through iterative replay. The skills evolve from the training corpus, which shares no trajectory with ReplayBench, and contain no evaluation-task text (Sec. 4.1; App. C.1, Eq. 10). Skill evolution extracts reusable rules and revises them in response to prediction errors, helping the simulator apply experience beyond individual examples.
[B. Controlled Ref./Skills Comparisons] Combined with Table 4, we provide comparisons across OpenCode, Claude Code, and Codex. Within each column, the backbone, harness, and generation-token limit are fixed; within each four-row group, Search + Coding is fixed. Gains in ( ) are relative to the corresponding baseline without Skills or Ref. (App. E.4, Table S3):
| Skills | Ref. | Search + Coding | OpenCode | Claude Code | Codex |
|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 54.0 | 55.4 | 55.9 |
| ✗ | ✓ | ✗ | 55.1 (+1.1) | 56.3 (+0.9) | 57.0 (+1.1) |
| ✓ | ✗ | ✗ | 56.8 (+2.8) | 58.2 (+2.8) | 59.0 (+3.1) |
| ✓ | ✓ | ✗ | 57.0 | 58.4 | 59.1 |
| ✗ | ✗ | ✓ | 55.6 | 57.1 | 57.8 |
| ✗ | ✓ | ✓ | 56.9 (+1.3) | 58.4 (+1.3) | 59.2 (+1.4) |
| ✓ | ✗ | ✓ | 58.2 (+2.6) | 59.5 (+2.4) | 60.3 (+2.5) |
| ✓ | ✓ | ✓ | 59.4 | 61.0 | 61.7 |
[C. What Skills Add] Adding Skills alone improves performance by 2.4–3.1 points, compared with 0.9–1.4 points from Ref. alone. Skills remain useful when reference examples are already available: adding Skills to Ref. yields a further 1.9–2.6 points across the six harness/tool settings. Their combination performs best under all three harnesses.
Evolving skills through iterative refinement provides substantial and useful environment knowledge, while reference examples provide complementary information. Skills also help with Search + Coding enabled: learned environment knowledge and agentic capabilities contribute together to an effective agentic world model.
[W3-2. Skill Transfer and Adaptation] What useful capability does skill evolution provide beyond supplying logged examples?
We provide further analysis of how learned environment knowledge can be reused across different models and updated as environments change.
[A. Reuse across Different Models] Our evolved skill repertoire can be reused by other models as agentic world models. The table compares different models with and without our evolved skill repertoire:
| Model | Backbone | + Our Evolved Repertoire |
|---|---|---|
| Qwen3.5-397B-A17B | 55.2 | 59.4 |
| Claude Sonnet 5 | 60.5 | 64.8 |
| Claude Opus 5 | 63.7 | 67.3 |
Evolved skills also benefit stronger models, including Sonnet 5 and Opus 5. The model that evolves the knowledge need not be the model that uses it. This indicates that ReplayWorld is a scalable approach.
[B. Updating Knowledge when the Environment Changes] After the OS changes from Ubuntu to Alpine, one replay cycle on 10 new traces raises the outcome-agreement advantage over the no-repertoire baseline from 4.5 to 15.0 points (Sec. 4.5; App. E.6, Table 15). Skill evolution supports both reuse and revision of environment knowledge: the simulator can retain useful rules and refine them when new behavior is observed.
Response to Reviewer UkF2 (1/4)
We sincerely thank you for the careful and constructive review and your thoughtful feedback on task generalization, benchmark reliability, and the value of skill evolution. We address each point below with clarifications and additional results.
Thank you for this thoughtful question. We have further clarified task-query construction in Appendix C.1 (Table S1) and added a similarity-filtered evaluation (Table S2).
ReplayWorld retains a +4.3-point advantage after highly similar queries are removed, compared with +4.2 on the full benchmark. This supports the robustness of our agentic world model to training–evaluation query similarity. The improvement persists when evaluation queries have no match at or above the specified similarity threshold.
Connection to Skill Evolution. Skill evolution samples from a large corpus at each round without exhaustively replaying it up to the selected round, so similar queries present in the corpus need not have contributed to the selected skills. The similarity-filtered evaluation directly checks the stability of this advantage.
[1] Xu et al. GraphWalker: Agentic Knowledge Graph Question Answering via Synthetic Trajectory Curriculum. 2026, Appendix B.2.
We would like to respectfully clarify that we have already provided OOD evaluation on another benchmark, AgentWorldBench, in the main paper (Sec. 4.2, Table 2) and appendix (App. E.1, Table 13).
We have compared ReplayWorld with LLM-as-WM (Table 2; App. E.1, Table 13); selected results are shown below:
The benefits of evolved skills extend beyond ReplayBench, including improved state consistency and multi-step simulation on other benchmarks and domains. Together with the query-similarity analysis, these results support the value of reusable environment knowledge across different evaluation tasks.