acceptodds
Under review as a conference paper at ICLR 2027

Agent World Modeling for Agent Test-time Scaling without Environment Replica

Abstract

Test-time scaling (TTS) improves large language model agents by sampling and selecting among multiple candidate trajectories, but common realizations assume that these trajectories can be executed in parallel. This assumption breaks in realistic deployments, where enterprise workspaces and cloud services are shared and persistent, costly or infeasible to replicate, and actions may introduce irreversible side effects or cross-trajectory interference. Existing generative world models avoid direct execution, but typically compress environment state into a single prompt, limiting their ability to reliably simulate transitions under partial observability. Instead, we introduce Agentic World Modeling (AWM), which reformulates next-observation prediction as a multi-turn evidence-acquisition task. For each hypothetical action, an agentic simulator adaptively queries the real environment through safe read-only interfaces, reasons over the returned evidence, and uses executable computation to construct the resulting observation without mutating shared state. Integrated with trajectory-level TTS, AWM enables parallel exploration over a single shared environment. Across -bench, AppWorld, and OfficeBench, AWM outperforms a single-shot simulation baseline by 40.1% on average, and surpasses a trained world-model baseline by 12.6% on average. These results establish interactive, read-grounded simulation as a practical foundation for safe and scalable agentic test-time scaling in shared, non-resettable environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
environment simulation, agentic test-time scaling, world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8L1ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies test time scaling for agents in environments that cannot easily be copied or reset. Running several candidate trajectories directly in such environments can cause unwanted changes and interference between trajectories.

The proposed Agentic World Modeling framework predicts action outcomes through an interactive process. For each hypothetical action, a simulator gathers relevant evidence using read-only tools. It can also use Python to construct the predicted observation. Context isolated subagents handle these prediction tasks. Several candidate trajectories are generated using the simulated observations. The highest-scoring trajectory is then selected for execution in the real environment.

The evaluation covers τ\tau-bench, AppWorld, and OfficeBench with three model backbones. The paper reports average improvements of 40.07% over SIMIA and 12.36 points over the trained world-model baseline where available. It also includes ablations for adaptive retrieval and Python execution. Further experiments study simulator-side scaling and token usage. A theoretical construction shows that adaptive retrieval can require exponentially fewer reads than non-adaptive retrieval when evidence is organized through a chain of dependencies.

Critical strengths
  1. The paper addresses an important limitation of agent test-time scaling. Many practical environments cannot support speculative writes from several parallel trajectories. Using read-only access to ground hypothetical rollouts is a useful direction. The distinction between retrieving evidence and reconstructing an observation is also sensible. Python execution can preserve exact document contents and structured values that an LLM might otherwise regenerate incorrectly. The theoretical analysis provides a clear example of why adaptive retrieval helps when each retrieved item identifies the next relevant item. The appendix explains that this is a worst-case separation rather than a guarantee for every environment.

  2. The empirical evaluation includes meaningful controls beyond the baseline comparison. AWM improves over both SIMIA and the trained world model across the reported settings in Table 1. Table 4 compares against a non-adaptive version with the same maximum read budget and Python capability. The token budget is also approximately matched. For GPT-5.4, adaptive retrieval improves AppWorld Level 3 success from 37.98% to 53.88%. OfficeBench 3 apps improves from 19.64% to 34.82%. These are substantial gains and provide direct evidence for adaptive evidence acquisition. Removing Python also consistently reduces performance. Finally, the reported task success is measured after execution in the actual benchmark environment. It is not based only on the simulator declaring its own trajectory successful.

Critical weaknesses
  1. Sec 3.1 states that the simulator reads from the unchanged initial environment. It infers the consequences of earlier hypothetical actions from the trajectory history. This is a central part of the method. Once a branch modifies or deletes an object, later reads from the real environment can conflict with that branch's state. The paper does not provide a concrete algorithm for reconciling these cases. Section 3.2 says that causal consequences are retained and mentions a TRACE interface. However, the representation of these consequences and the behavior of TRACE are not explained in enough detail to reproduce the process. This becomes particularly important when subagents return only compact results.

The evaluation in Appendix C.3 does not fully test this issue. It appends real observations to the history at every step. This is useful for measuring one-step prediction quality, but it avoids error accumulation across simulated transitions. All six qualitative examples also concern read operations. They show that retrieving real records helps recover missing information. They do not demonstrate consistency after several hypothetical writes. The end-to-end results provide indirect evidence on stateful tasks, but a direct analysis of read-after-write and write-after-write dependencies is still needed. The reported 41.1% prediction accuracy on AppWorld also makes this analysis important.

The final execution protocol is another unresolved detail. The paper says that the selected trajectory is committed to the real environment. It does not clearly state whether this means replaying a fixed action sequence or allowing the agent to revise its actions using real observations. These choices can lead to very different outcomes when simulation is imperfect. The issue is especially important for τ\tau-bench because future user responses may differ from the simulated dialogue. Without this information, it is difficult to assess how the proposed simulation supports reliable execution.

  1. The SIMIA case studies repeatedly show that the fixed context omits the exact file or database records needed for the requested action. Appendix B.4 states that the baseline context constructor can access the same underlying information. However, the actual retrieval and serialization procedure is not described in enough detail to assess its quality. The large headline gain therefore combines the benefit of obtaining relevant evidence with any benefit from better simulation. Table 4 is a much stronger control and does support adaptivity. Still, the paper needs clearer baseline construction details before the 40.07-point headline improvement can be interpreted properly.

The simulator is also used to score completed trajectories. The reported results therefore combine changes in trajectory generation and trajectory ranking. They do not show how much of the gain comes from each. At BoN@1 there is no choice between candidates, yet large improvements are already present. This supports better simulated rollouts, but those gains should not be described only as better trajectory selection. A fixed-candidate evaluation with a common scorer would help separate these effects.

Several evaluation details are missing. Appendix C.3 reports observation prediction accuracy without specifying the matching rule. It is unclear whether this uses exact matching, normalized structured matching, or model-based judgment. The same section says that tasks are sampled from each benchmark but only reports accuracy for AppWorld and OfficeBench. The trajectory-scoring rubric and the aggregation of multiple simulator predictions are also not sufficiently described.

The efficiency evidence is narrower than the practical claims. Token usage is measured on 64 OfficeBench tasks. The rollout budget and simulator aggregation setting for this comparison are not clearly stated. There is no corresponding analysis of latency or environment-query overhead. The subagent ablation reports token savings but does not report whether prediction quality is preserved. These omissions matter because the method introduces several dependent simulator calls for each action.

Questions for authors
  1. How are hypothetical state changes represented and passed between prediction subagents? Can the authors show a complete example with several writes followed by reads. How does the simulator resolve a conflict between a branch's hypothetical state and the unchanged real environment? What exactly does TRACE return?

  2. How is the selected trajectory executed? Is it replayed as a fixed action sequence or does the agent replan using real observations? For τ\tau-bench, how are simulated user responses handled when the real user simulator responds differently? Please provide the rollout and execution procedure.

  3. What matching rule is used for observation prediction accuracy in Appendix C.3? Is the simulator restricted to the pre-action environment state when predicting a mutating action? Can you report results separately for reads, writes, and reads that depend on earlier hypothetical writes?

  4. How are the fixed contexts for SIMIA and WM-Training constructed? Why do the examples omit files explicitly named in the action? Please provide the retrieval procedure and prompts. Also clarify which model acts as the task policy in the WM-Training comparison and whether the policy is held fixed across simulator variants.

  5. How are completed trajectories scored? Does the scorer use additional read-only queries? How are the M independent predictions aggregated? Can you separate trajectory-generation quality from ranking quality using a common candidate set or a common scorer?

  6. What N and M settings are used for the token-cost experiment? Does the accounting include all prediction and verification subagents? Please report environment-query counts and wall-clock latency. It would also help to report task success for the subagent ablation and uncertainty estimates for the main controlled comparisons.

Reviewer_ya5jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies test-time scaling (TTS) for LLM agents in shared, persistent, non-replicable environments (e.g., enterprise workspaces, cloud services), where the standard assumption of forking/resetting the environment breaks down and speculative execution of candidate trajectories would cause irreversible side effects and cross-trajectory interference.

Critical strengths

S1. Addresses a real and underexplored bottleneck in agentic TTS. Most agentic TTS work assumes resettable/replicable environments. The setting here — a single shared, non-resettable environment where candidate trajectories cannot be safely executed — is genuinely common in deployment and has received little attention. The problem statement is crisp and the POMDP formalization is appropriate.

S2. The core idea is simple, well-executed, and clearly differentiated from prior world models. The key insight — that read-only access is both safe and parallelizable, so the simulator itself can interactively interrogate the real environment to ground each hypothetical transition — is elegant. It converts world modeling from "generation under a fixed information set" to "evidence-driven simulation." The auxiliary-POMDP formulation (§3.1) and the read–compute interface (§3.2) are clean and reproducible in principle.

S3. Comprehensive and convincing empirical results. Three complementary benchmarks (tool–agent–user dialog, API/code ecosystems, office automation), three simulator backbones, Best-of-N from 1 to 16, and three baselines spanning the two existing world-model paradigms (prompting-based SIMIA, training-based WM-Training). Winning essentially every comparison (Table 1) is strong evidence. The direct next-observation prediction accuracy study (§C.3: 41.1% / 82.2% for AWM vs. 6.87% / 5.94% for SIMIA) helpfully isolates the mechanism behind the end-task gains.

S4. Thoughtful ablations that map onto the paper's claims. The non-adaptive AWM ablation (§C.4) is the most valuable: it holds the read budget, Python capability, and token budget approximately fixed and removes only sequential adaptivity — and the gap grows with task compositionality (e.g., AppWorld L1→L3, OfficeBench 1App→3Apps), exactly as the adaptivity story predicts. The subagent ablation (9.37x / 8.84x token inflation without context isolation, Fig. 4) addresses the natural cost concern.

S5. Honest case studies. The qualitative examples (Figs. 5–7) show SIMIA fabricating empty directories / zero transactions / nonexistent-file errors when evidence is omitted, while AWM recovers the actual records; residual AWM errors (ordering, formatting, missing headers) are reported rather than hidden.

Critical weaknesses

W1. The theory (Prop. 3.1) is a worst-case separation on a contrived family, and the framing in the main text risks over-selling it. The D-stage K-ary lookup construction with uniform W⋆ is a clean adversarial family, and the mutual-information + Fano argument is correct as far as I can verify. However: (a) the lower bound holds only for this specific construction, not for τ-bench/AppWorld/OfficeBench; (b) the constant gap "M_NA ≥ (1−δ−1/β)·K^D" is vacuous for realistic β (e.g., β = 100 bits) unless δ is extremely close to 1; (c) the appendix concedes it is "a worst-case separation rather than a claim that every environment exhibits an exponential gap," but the abstract/introduction lean on the exponential claim. I would ask the authors to soften the main-text framing or add a paragraph in §3.3 making the worst-case nature explicit (the appendix does this, but main-text readers may not go there).

W2. Potential evaluation asymmetry between AWM and the baselines. §B.4 states the baselines' context constructor can access the same records as AWM. But two asymmetries remain: (a) Compute/tokens. AWM's per-transition evidence acquisition is a multi-turn agentic process with subagents and Python execution; SIMIA is a single forward pass. Figure 4 reports token costs for only 64 OfficeBench tasks and only against baselines at one setting; per-task latency and wall-clock cost (including tool-call round trips) are not reported at all. Since the paper's pitch is practical TTS, a latency/cost-vs-accuracy comparison is important. (b) Read budget. The appendix says reads are capped at six per transition — but is SIMIA's context construction also capped at six records? §C.4's "approximately matched token budget" is the right control, but Table 1's headline comparison does not clearly state whether the same budget applies. Please clarify.

W3. The "world model" here is partially the environment itself; the contribution boundary should be discussed more carefully. Because AWM reads the true environment state during simulation, it is closer to a grounded simulator wrapper than an autonomous world model. In particular, the oracle "Real-env BoN" ceiling in Table 3 shows AWM(M=4) still trails it substantially (e.g., AppWorld/GPT-5.4: 77.78 vs. 93.68 at N=16; AppWorld/Qwen: 69.40 vs. 81.37), so simulation error remains the binding constraint. A discussion of when AWM's advantage vanishes — e.g., environments where reads are expensive/rate-limited or where the transition genuinely depends on latent state not exposed by any read-only interface (other agents' concurrent writes, stochastic environment internals) — would strengthen the paper. The current claim "interactive, read-grounded simulation as a practical foundation" would benefit from stating its scope.

W4. Success criterion for trajectory selection is under-specified and potentially circular. §B.3 says the simulator scores the predicted outcomes and the highest-scoring candidate is selected, and that "we use the agent itself to predict observations and assess trajectory correctness." Using the same backbone to both generate and verify is a known self-preference bias risk. How exactly is the score computed (LLM-as-judge? binary predicted success? value model?)? How often does the selector pick a trajectory whose predicted success is wrong (selection error rate vs. simulation error rate decomposition)? This is a central component of the TTS loop and deserves its own analysis.

W5. Missing baselines and comparisons. (a) No comparison to tree-search / lookahead methods with rollback in resettable benchmarks (e.g., a trajectory-tree search with simulated reversal) to quantify how much is lost by forbidding replication. (b) No comparison to simply letting the task agent itself perform read-grounded planning without the world-model framing (e.g., a "think-before-act with reads" agent at the same read budget) — this would test whether the hierarchical AWM machinery is necessary or whether read access alone suffices. (c) No combination AWM + WM-Training; §4.2 claims adaptive grounding "can provide a stronger basis than a separately trained transition model," but the natural question is complementarity.

W6. Reproducibility caveats. The three backbones include two closed, fast-moving API models; the trained baseline is a single 35B-A3B checkpoint, so the "trained world model" comparison is drawn from one point in the design space. Also, some referenced systems (SIMIA, WM-Training) are contemporaneous arXiv work — reviewers cannot easily verify that the baselines were tuned as carefully as AWM (e.g., SIMIA's context-construction heuristics, the choice of what to include in its fixed prompt). A sensitivity study on SIMIA's context size/coverage would help rule out a weak-baseline effect.

W7. Minor writing issues. The paper is generally well written but has typos and formatting artifacts: "adaptive our appraoch" (§3.3, line ~247), a doubled ";;" after "tool-use capabilities" (§4.1), Figure 3's y-axis label rendered as "DropPython" fragments, and the τ-bench task counts in §B.1 (50 airline / 114 retail) vs. the pooled "164 for τ-bench" in Table 3 should be cross-checked for consistency (they do sum, but stating the pooling convention earlier would avoid confusion).

Questions for authors

(Budget parity in Table 1) Is the number of environment records/reads available to SIMIA's fixed context construction matched to AWM's six-read cap in the main results? If not, could you report SIMIA with a matched (even enlarged) evidence budget? (Selection mechanism) What exact scoring function selects the best-of-N trajectory, and how sensitive are the results to it? Have you measured the selector's accuracy independent of the world model? (Latency) What is the wall-clock cost per task for AWM vs. SIMIA / WM-Training / Real-env BoN? Multi-turn evidence acquisition with subagents multiplies sequential tool round-trips; is the method practical for interactive latency budgets? (Scope) How does AWM degrade when (a) read-only interfaces expose only partial state (e.g., another user concurrently mutates the environment mid-rollout, so s0 drifts), or (b) transitions depend on stochastic internals not observable through reads? Does the auxiliary-POMDP formulation assume s0 is frozen during the auxiliary episode even in the real system? (Theory) Can you state in the main text that Prop. 3.1 is a worst-case separation for a specific construction, and comment on whether any of the evaluated benchmarks provably (or empirically, via the C.4 ablation gaps) instantiate the K^D dependence structure? (Combination) Does AWM + WM-Training (trained model proposing reads, or trained model + read grounding) further improve performance? This would clarify whether the two paradigms are substitutes or complements. (τ-bench observation space) τ-bench observations include user replies generated by an LLM-simulated user. How does AWM predict these — are they also read-grounded, or does it hallucinate user behavior? The paper does not discuss user-side simulation at all.

1 comment

Dear Reviewer ya5j,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Reviewer_92MEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets agentic test-time scaling (TTS) in environments that cannot be forked or reset (e.g., shared enterprise workspaces), where executing multiple candidate trajectories in parallel is unsafe because mutating actions cause cross-trajectory interference. Existing LLM world models avoid execution but predict the next observation in a single shot from a fixed, lossily compressed context, which omits relevant evidence in large environments. The paper proposes Agentic World Modeling (AWM), which recasts next-observation prediction as a multi-turn evidence-acquisition process: for each hypothetical action, a context-isolated subagent adaptively issues read-only queries (READ/LIST/CALL/TRACE) to the real environment, reasons over returned evidence, and uses restricted Python execution to deterministically construct the predicted observation.

Critical strengths
  1. The tension between parallel trajectory sampling for agentic TTS and non-replicable, persistent environments is real and underexplored; most agentic TTS work implicitly assumes resettable sandboxes. The cross-trajectory interference example (Figure 1, top-left) is clear and compelling.

  2. Clean, intuitive reformulation. Turning next-observation prediction from single-shot generation into an interactive POMDP with a read budget is natural, and the four-tool read-only interface (READ/LIST/CALL/TRACE) plus executable prediction (Python as a deterministic "decoder") is a practical instantiation. The context-isolated subagent design is a sensible engineering solution to nested-context blow-up.

  3. Theoretical support matches the mechanism. The adaptive-vs-non-adaptive separation is proved correctly via Fano's inequality for a D-stage dependency chain, and the paper honestly scopes it as a worst-case construction. The empirical non-adaptive ablation (Table 4) shows gaps that grow with task complexity (AppWorld L1→L3: 5.43→15.89 pp), which is the qualitative pattern the theory predicts.

  4. The paper includes an oracle ceiling (real-environment BoN in Table 3) and direct one-step fidelity numbers rather than reporting only downstream success.

Critical weaknesses
  1. The experiments never test the paper's motivating regime, and the "read-only" realization appears to access simulator-internal ground-truth state. All three benchmarks are fully resettable, inspectable local sandboxes. The precise setting the paper argues is unrealistic. More importantly, the case studies show the subagent reading the simulator's raw state tables directly. This is not the same as the agent-facing read-only API surface: it is privileged access to the ground-truth database whose deterministic projection largely is the environment's transition/read logic.

  2. The showcased evidence is almost entirely for read actions which are the easy case where no world model is needed. All six case studies (Figures 5–7) predict observations for read actions. But a read action is safe by definition: in the paper's own framework, candidate trajectories can simply execute read-only actions against the shared environment (Section 3.1 argues reads are safe and parallelizable). Predicting a read's result by performing the read is therefore not a world-modeling capability; it is direct execution. The paper's entire motivation rests on mutating actions (delete, create, update, refund) whose outcomes depend on server-side validation, policies, permissions, cascading effects, and generated fields.

  3. The main table omits the obvious reference point, direct execution without simulation, and the appendix data shows simulation hurts at N=1.

  4. Absolute next-observation fidelity is low on AppWorld, and the metric is underspecified.

Questions for authors

What fraction of predicted transitions in each benchmark are mutating actions? What is one-step next-observation accuracy separately for read and write actions, and what does AWM do when a write's outcome depends on server-side validation (policy checks, permission errors, conflict responses)?

1 comment

Dear Reviewer 92ME,

Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes this Wednesday, November 18. If you have any remaining questions, we'd be happy to address them before the deadline!

Best regards,

The Authors

Comments on the paper

1 comment

Dear Reviewers,

Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.