acceptodds
Under review as a conference paper at ICLR 2027

World Boundary Bench: Probing the Failure Mode of Language World Models

Abstract

Language World Models (LWMs) are evolving from next state decision reference to interactive world simulators. Their reliability depends on whether they could maintain decisive world knowledge, preserve the consistent states in long horizon, and maintain their simulator role when environment data contains instruction like text. Therefore, to further investigate current LWMs' failure modes, we introduce World Boundary Bench (WBB), a benchmark spanning Terminal, MCP, and Web environments. WBB contains three tracks: 1) Causal Forking Track tests whether an LWM notices a small but decisive difference, such as whether incompatible torch and torchvision versions cause pip installation to fail. 2) World Commitment Track simulates scenarios with limited context and observes what kind of next state the LWM will output, and whether its further outputs remain consistent with that state. 3) Role Boundary Track tests whether embedded instructions within terminal or tool commands detach the LWM from its world model role. Across 13 proprietary and open source models, Kimi K3 leads Causal Forking and Role Boundary and ties with Qwen 3.8 Max 0902 on World Commitment, while models exhibit distinct failure modes across tracks. In our sim to real study, rejection finetuning on LWM self-reported success trajectories improves real environment success from 2.81% to 4.41% after simulated practice, illustrating potential agentic training utility beyond simulation failures.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Language World Model, Agent, Agentic Training, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cuovOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces World Boundary Bench to evaluate language models as environment simulators. Its three tracks test sensitivity to decisive input changes, state consistency across interactions, and resistance to instruction-like environment data. It evaluates 13 models and includes real-environment replay and exploratory agent fine-tuning experiments.

Critical strengths

The controlled pairs and executable checks provide a concrete way to diagnose simulation errors. Allowing models to establish an initial state and then checking subsequent consistency is a useful evaluation design. The real-environment replay also provides direct evidence that simulated success can be misleading.

Critical weaknesses
  1. The additional insight from the benchmark is limited. The main findings—models overlook decisive details, lose state consistency, and follow instructions embedded in data—are plausible failure modes. Quantifying them is useful, but the paper does not sufficiently establish what these measurements tell us about choosing or improving a simulator. In particular, it does not show whether WBB scores predict real-replay success or downstream training benefit.

  2. The training experiments do not resolve the reliability concerns motivating the paper. They show small improvements from a fixed set of selected trajectories on ToolSandbox, but do not examine whether continued optimization encourages agents to exploit simulator errors. The Role Boundary track tests fixed instruction-like inputs, rather than agents learning to obtain apparent success through simulator weaknesses. The two downstream configurations also change both the agent and simulator, preventing a controlled assessment of how simulator quality affects learning.

Questions for authors
  1. Do WBB scores, or particular error categories, predict downstream replay success or training gains when the agent is held fixed?

  2. Does repeated training against a simulator increase the gap between simulated and real success? Can the proposed benchmark detect simulators that are particularly vulnerable to such exploitation?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thanks for your valuable review and suggestions. Below we respond to Weaknesses (W) and Questions (Q). We use [Added] to indicate we add and update the corresponding content about the question or weakness in the main revised paper with blue color.


W1&Q1: The benchmark's additional insight and value for choosing or improving simulators are insufficiently established. It’s unclear whether WBB scores predict real-replay success or downstream training benefit.

  • Thanks for this important question. First, WBB gives insights on where simulators fail and whether training improves these specific capabilities and is a benchmark used to explore the failure boundaries of a simulator through carefully constructed controlled tests. The test conditions and their distribution emphasized in WBB are not the same as those for a specific downstream task. Therefore, it cannot be directly expected that the overall WBB score will rank consistently with the success rate or training gains of a particular downstream task.

  • For example, Qwen-AgentWorld improves Causal Forking joint accuracy over its base model from 22.0% to 27.0%, while World Commitment remains at 54.0%. Thus, better causal prediction does not automatically mean better state consistency. These separate measurements help users compare simulators for their intended tasks and identify weaknesses to target in further training.


W2(a)&Q2(a): Does repeated optimization encourage and widen the simulated real success gap? [Added]

  • Thanks for this important question. We conducted three rounds of iterative training with Qwen3-4B-Instruct-2507 and a fixed Kimi K3 simulator. Each round collects new trajectories, selects all self-reported successes, and continues SFT from the previous checkpoint. We replay all 1,032 trajectories in the real environment and evaluate each updated checkpoint on the same eval tasks.
RoundSimulated successReal-replay successGap (pp)Real success after SFT
144.38%16.18%28.203.88%
253.39%16.57%36.824.95%
360.27%16.28%43.995.61%
  • This experiment reveals a clear distinction between simulated success and actual learning progress. Across three rounds, simulated success rises from 44.38% to 60.27%, while real-replay success remains around 16%. Meanwhile, real-environment performance after SFT increases from 3.88% to 5.61%.

  • These additional results extend our investigation of the two questions in Section 4. First, regarding “Is success in a simulated world real?”, self-reported success increases substantially while real replay success remains approximately unchanged, widening the gap from 28.20 to 43.99 percentage points. Thus, repeated optimization does not necessarily improve the real execution validity of simulated trajectories, even as their apparent success increases. Second, regarding whether LWM self-selected trajectories lead to serious degradation, we observe no degradation over the three training rounds: despite including trajectories that fail real replay, the self selected training data yields checkpoints whose real environment success increases from 3.88% to 5.61%. Together, these findings show that, in this setting, learning gains from self selected simulation data can coexist with an increasing simulation reality gap.

  • We have complemented this new experiment and content in our main context in Lines 467-491.


W2(b): The Role Boundary track tests fixed instruction-like inputs, rather than agents learning to obtain apparent success through simulator weaknesses.

  • Thanks for this question. First, we are testing the Language World Model, which is a world simulator, rather than an Agent. LWM is a world simulator that responds to the agent's action with concrete world modeling results.
  • Second, the Role Boundary Track also has practical grounding: as illustrated in Lines 463-465 and Lines 1491-1494, in a Qwen-AgentWorld rollout on the SWE-Gym instance moto-5513, the LWM proposes actions such as running grep and inspecting the code, instead of returning the environment observation produced by the agent’s action. The model therefore switches from simulating the environment to behaving like the agent, matching the failure tested by the Role Boundary Track.
  • Third, WBB also has practical downstream grounding: our downstream ablation identifies 488 distinct self accepted but replay failed trajectories containing failure patterns consistent with WBB across ToolSandbox, MCPMark, and SWE-Gym (Lines 463-465). These findings show that the tested failures also occur in practical simulator use.
AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2(c): Changing both the agent and simulator confounds the effect of simulator quality on learning.

  • Thanks for this question. We complement an new ablation study that using the same agent: Qwen3-4B-Instruct-2507 and Kimi K3, which achieves higher WBB scores, as a comparison to the (Qwen3-4B-Instruct-2507, Qwen-AgentWorld) old pair. The results are shown as following:
SimulatorReal replay successReplay verified RFT success
Qwen-AgentWorld15.60%4.01%
Kimi K316.18%4.28%

This comparison removes the change in agent and directly evaluates whether trajectories generated through interaction with each simulator succeed under real execution.


Q2(b): Can the proposed benchmark detect simulators that are particularly vulnerable to the enlarging gap between sim and real? [Added]

  • Thanks for this question. To investigate this, we additionally conducted three rounds of training with DeepSeek V4 Flash, keeping the initial agent and training procedure consistent with the Kimi K3 experiment.
SimulatorRound 1 gapRound 3 gapGap increase
Kimi K328.20 %43.99 %+15.79 %
DeepSeek V4 Flash22.97 %42.64 %+19.67 %

Both simulators showed a widening gap between the simulation success rate and the actual playback success rate during three rounds of training, indicating that this phenomenon also occurred in another set of simulator configurations. We have revised the main context and complemented this ablation study in Lines 492-504.

Reviewer_cuovOpenReview ↗

Response to the authors' rebuttal

We thank the authors for a substantive and well-executed response. The three-round iterative training experiment (W2(a)) directly addresses our original question and is well designed: the finding that simulated success rises while real-replay success stays flat, with the gap widening from 28.20 to 43.99 percentage points, is an interesting and carefully demonstrated result — learning gains can coexist with increasingly unreliable simulated success signals. We acknowledge the value of this addition.

However, the additional experiments offered to connect WBB scores to simulator quality raise new concerns rather than resolving our first weakness (W1).

The controlled simulator comparison is within noise. With the agent fixed (Qwen3-4B-Instruct-2507), the difference between Kimi K3 and Qwen-AgentWorld is 0.58 percentage points in real-replay success (16.18% vs. 15.60%) and 0.27 points in replay-verified RFT (4.28% vs. 4.01%). The latter is comparable to the paper's own reported RFT seed standard deviations (±0.13–0.39 across three seeds). No variance or significance is reported for any of these comparisons. Two simulators that differ enormously in WBB scores — Kimi K3 is the strongest overall model evaluated, while Qwen-AgentWorld is among the weakest open-source models — produce downstream benefit differences that are statistically indistinguishable from zero. This is the cleanest available test of the predictive value of WBB scores, and the outcome does not support it.

The gap-growth comparison is metric-dependent and based on n = 2. In Table 2, the conclusion rests on the increase in the gap (+15.79 for Kimi K3 vs. +19.67 for DeepSeek V4 Flash) across two simulators. Yet at round 1 the ordering is reversed: the higher-WBB simulator (Kimi K3, 28.20%) exhibits a larger initial gap than the lower-WBB simulator (DeepSeek V4 Flash, 22.97%), and the final gaps at round 3 are nearly identical (43.99% vs. 42.64%). With two data points, and a conclusion that depends on whether one examines the initial gap, the final gap, or the growth, this comparison cannot support the claim that WBB scores predict exploitation vulnerability. A reasonable confound also remains: the two simulators differ in every dimension, so any difference in gaps cannot be attributed to the boundary performance that WBB measures.

Our position. Our original first weakness was that the paper does not establish what WBB measurements tell us about choosing or improving a simulator. The evidence offered in the rebuttal — a qualitative audit of 488 trajectories, a statistically indistinguishable difference in the controlled comparison, and a metric-dependent n = 2 comparison — does not change this assessment. We note that the paper's most robust practical finding, that self-reported simulated success severely overestimates real-replay validity, follows from the replay-verification protocol itself and remains accessible without WBB. Our assessment is therefore that the paper contains a carefully executed and useful negative result about simulator reliability, but its central contribution — that the benchmark scores carry diagnostic and predictive value — remains unestablished, with the strongest controlled evidence currently pointing against it. Our rating is unchanged.

Reviewer_vzxaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

WBB tests language world models on causal changes, consistency with their chosen initial states, and instructions embedded in environment data. The paper evaluates 13 models, checks simulated behavior through real replay, and includes a small fine-tuning study.

Critical strengths

Letting the model choose an initial state and then checking consistency with that choice handles underspecification well. The paired causal tests also make guessing a common outcome less effective. The gap between simulated success and real replay is a useful finding, supported by a human audit of the verifier. The executable checks give the benchmark a firmer basis than asking another model whether a simulation looks plausible. In particular, a model must start from a feasible state; repeating an invalid initial choice is not enough to pass.

Critical weaknesses

The 100 causal pairs come from five mechanisms. Appendix C.1 varies names, values, and histories but keeps the transition rule. I would put more weight on performance across these families than on the number of instances. It is still unclear how far the findings extend beyond these rules.

The fine-tuning gains are small, at 1.60 and 1.07 points. Each condition uses 150 trajectories, but the number of supervised tokens may differ. That makes the training result harder to interpret. The paper already supplies subtask breakdowns. My concern is the unit of evidence: many variations of one rule do not establish the same breadth as additional rules. For downstream training, I would keep the claim exploratory until the token-budget comparison is resolved.

Questions for authors

Can you give results and uncertainty by rule family? How many assistant tokens does each training condition contain, and does matching that budget preserve the gain? The existing breakdowns could help show whether aggregate failures are widespread or concentrated in a few mechanisms.

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thanks for your supportive review and suggestions. Below we respond to Weaknesses (W) and Questions (Q). We use [Added] to indicate we add and update the corresponding content about the question or weakness in the main revised paper with blue color.


W1: The causal pair instance variation within five causal mechanisms does not establish broad rule coverage.

  • Thanks for your valuable review. We agree that a broad rule coverage is very important, so our WBB spans different domains (e.g., Terminal, MCP, and Web...) and three complementary tracks, with 600 evaluation instances, and more than 30 rules across the three tracks, which is comparable to existing well-known benchmarks, such as Terminal-Bench 2.0 [1], and OSWorld [2] that contain about 89 and 369 instances.
  • Beyond the instance numbers, WBB emphasizes controlled, executable tests of specific simulation failures. That’s also the reason why we choose terminal, MCP and other tasks that can be verified with ground truth answers. We agree that broader coverage would strengthen the benchmark and thank again for this constructive advice. We will further extend WBB to more domains in our future work.

[1] Terminal-Bench 2.0 (https://arxiv.org/abs/2601.11868).

[2] OSWorld (https://arxiv.org/abs/2404.07972).


W2(a): The fine-tuning gains are small, so downstream claims should remain exploratory. [Added]

  • Thanks for this review. The main purpose of the finetuning ablation is to investigate two questions: 1. Is Success in a Simulated World Real? 2. Will LWM Self Selected Failed Trajectories Lead to Serious Degradation? Therefore, the concrete ablation includes different domains and different LLM, LWM pairs for this investigation.
  • We show that the simulated success could be unsuccessful in real replay and the self-selected failed trajectories could even improve the agentic performance, which provides a simple improvement method.
  • We complemented several new ablation studies on whether repeated optimization encourage the sim-real success gap in Lines 475-511. In details, We conducted three rounds of iterative training with Qwen3-4B-Instruct-2507 and a fixed Kimi K3 simulator. Each round collects new trajectories, selects all self reported successes, and continues SFT from the previous checkpoint.
RoundSimulated successReal-replay successGap (pp)Real success after SFT
144.38%16.18%28.203.88%
253.39%16.57%36.824.95%
360.27%16.28%43.995.61%

This experiment reveals a clear distinction between simulated success and actual learning progress. These findings show that, learning gains from self selected simulation data can coexist with an increasing simulation reality gap, which also extend our previous experiments.


Q1(a)&Q1(c): Report results and uncertainty by rule family to show the concentration of failures.

  • Thanks for this question. We have reported the detailed results by rule family in our Appendix Lines 1055-1133, which contains each subtask results of WBB. We can see that different LLM exists in different failure modes when serving as Language World Models. We also give analysis in Lines 369-401.

W2(b)&Q1(b): The number of supervised tokens in the finetuning process may differ, which may make the gain not persist under a matched token budget.

  • Thanks for this question. Our experiment compares the training utility of trajectories obtained through different collection and selection procedures under the same trajectory budget and SFT configuration. Therefore, matching the number of trajectories preserves the unit of comparison: complete task interactions. Differences in trajectory length are part of the data produced by these procedures.
  • Token matching would require changing the number of trajectories, truncating interactions, or adjusting training exposure, thereby changing the comparison. We therefore retain the trajectory matched design.
Reviewer_qNuUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces World Boundary Bench (WBB), a diagnostic benchmark for studying failure modes in language world models (LWMs) used as interactive environment simulators. WBB has three tracks: Causal Forking, World Commitment, and Role Boundary, that test different capabilities required for reliable world simulation. It covers Terminal, MCP, and Web environments, and evaluates 13 proprietary and open-weight models. The paper also presents a sim-to-real study showing that LWMs often report success for trajectories that fail when replayed in real environments. Reject finetuning on trajectories that the LWM itself reports as successful yields modest improvements in agent performance in the tested setting.

Critical strengths
  1. The three focused tracks test complementary aspects of LWM simulation fidelity and provide a useful diagnostic framework.

  2. The results show distinct failure profiles across tracks, indicating that an aggregate score could hide important weaknesses.

  3. The real-environment replay analysis is a useful check on whether simulated success is a reliable indicator of actual task success.

Critical weaknesses
  1. The benchmark is relatively small: it contains 100 pairs for Causal Forking, 200 episodes for World Commitment, and 100 pairs for Role Boundary. This limits the comprehensiveness of its results. The title (world boundary benchmark) and framing could give the impression of broader coverage than the benchmark provides.

  2. The paper should clearly distinguish its contribution from prior probing studies of LLMs as world models. For example, R-WoM (Mei et al., 2026) probes next-state prediction using lexically similar distractors in the computer-use domain, which is relevant to the Causal Fork track.

  3. For Causal Forking, it is unclear how the evaluation separates action-conditioned simulation from parametric recall of environment-specific facts. The paper could better explain why the tested facts are representative of the knowledge LWMs need to simulate (e.g., torch / torchvision dependencies), and how much of the task depends on knowledge that an LWM may not have encountered during training.

  4. The evaluation combines programmatic checks, rubric-based judging, and human review. Although the paper reports agreement with human evaluations, the review process should be made clearer in the main text.

  5. The fine-tuning result is exploratory and only covers a single task (ToolSandbox).

  6. Role Boundary is a more general problem on instruction following vulnerability and prompt injection. It's unclear why this is special for LWM simulation.

Refs:

  • Mei et al., 2026. Retrieval-augmented World Model For Computer-use Agents.
Questions for authors
  1. During real environment replay, how is a trajectory scored if the simulated state diverges from the real state and a saved action is no longer valid? Does such invalid action count as a replay failure?
10 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thanks for your valuable review and suggestions. Below we respond to Weaknesses (W) and Questions (Q). We use [Added] to indicate we add and update the corresponding content about the question or weakness in the main revised paper with blue color.


W1: The benchmark is small, limiting the comprehensiveness of its findings.

  • Thanks for your valuable review. First, existing well-known benchmarks, such as Terminal-Bench 2.0 [1], and OSWorld [2] contain about 89 and 369 instances. Comparing with them, WBB contains 600 evaluation instances, with different domains (e.g., Terminal, MCP, and Web...) and three complementary tracks. We think much instances may increase testing costs, so we have reduced the number of our benchmarks to the current state to balance the cost and the effect.
  • WBB emphasizes controlled, executable tests of specific simulation failures. We agree that broader coverage would strengthen the benchmark and we will further extend WBB to more domains, task mechanisms in the future work.

[1] Terminal-Bench 2.0 (https://arxiv.org/abs/2601.11868).

[2] OSWorld (https://arxiv.org/abs/2404.07972).


W2: Clarify the contribution beyond prior LLM world-model probing studies, especially R-WoM. [Added]

  • Thanks for this review. We have clarified the difference between our work and other related works in the Related Work Section (Lines 1208-1227) and in Table 8 (Lines 1245-1259).
  • As for the R-WoM paper, we thank the reviewer for your kind reminder about this work, and here is the main difference:
  1. R-WoM’s next-state probe asks models to select the correct observation against a lexically similar distractor, which focuses on the output. Different from it, our Causal Forking changes one decisive input and requires the model to generate correct observations for both resulting branches.
  2. R-WoM evaluates long-horizon plans against reference procedures. Our World Commitment allows multiple valid initial states and checks whether subsequent interactions remain consistent with the state the model itself established.
  3. Our Role Boundary additionally tests whether instructions embedded in environment data cause the model to abandon its simulator role.
  • We have cited and discussed R-WoM in our Related Work Section in Lines 1237-1240.

W3: The paper could better explain why the tested facts are representative of the knowledge LWMs need to simulate (e.g., torch / torchvision dependencies), and how much of the task depends on knowledge that an LWM may not have encountered during training. [Added]

  • Thanks for this important question. Our tests in Causal Forking Track include established package versions such as torch 2.2.0, as well as controlled offline packages whose dependency requirements and available versions are explicitly provided in the interaction history.

  • To resolve this concern, we directly ask the tested 7 frontier models about the established package’s versions. The 7 tested froniter models could successfully distinguish the direct conflicts between the package’s versions, while some of them fail when playing the LWM role.

  • As for the offline packages, we expose the required constraints to the tested LWM. For example, rawb-plugin-alpha requires rawb-core>=2,<3, with versions 2.4.0 and 3.1.0 available locally. Installation succeeds when rawb-plugin-beta also requires rawb-core>=2,<3, but fails when beta instead requires rawb-core>=3,<4. These constraints are explicitly visible to the model.

  • Therefore, our Causal Forking Track is indeed testing the world modeling ability about the causal. Dependency compatibility is relevant to LWM simulation because it determines whether installation succeeds and what software can subsequently execute.

  • We thank again for raising such an important question. We revise the main context and clarify the distinction between required background knowledge and instance specific information in the main context in Lines 150-157.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W4: Clarify the evaluation and human-review process in the main text.

  • Thanks for this review. We clarify the human evaluation process in Section 2.5 (Lines 291-295, 326-328) and more details in the Append part (Section F.5) due to the page limitation.

W5: The finetuning ablation is exploratory and limited. [Added]

  • Thanks for this review. The main purpose of the finetuning ablation is to investigate two questions: 1. Is Success in a Simulated World Real? 2. Will LWM Self Selected Failed Trajectories Lead to Serious Degradation? Therefore, the concrete ablation includes different domains and different LLM, LWM pairs for this investigation.
  • We show that the simulated success could be unsuccessful in real replay and the self-selected failed trajectories could even improve the agentic performance, which provides a simple improvement method.
  • We complemented several new ablation studies on whether repeated optimization encourage the sim-real success gap in Lines 467-504. In details, We conducted three rounds of iterative training with Qwen3-4B-Instruct-2507 and a fixed Kimi K3 simulator. Each round collects new trajectories, selects all self reported successes, and continues SFT from the previous checkpoint.
RoundSimulated successReal-replay successGap (pp)Real success after SFT
144.38%16.18%28.203.88%
253.39%16.57%36.824.95%
360.27%16.28%43.995.61%

This experiment reveals a clear distinction between simulated success and actual learning progress. These findings show that, learning gains from self selected simulation data can coexist with an increasing simulation reality gap, which also extend our previous experiments.


W6: Why is Role Boundary specifically relevant to LWM simulation rather than general prompt-injection vulnerability?

  • Thanks for this question. We agree that Role Boundary is not only relevant to LWM, but also general LLM prompt-injection, but we think it’s more essential for the LWM, because:
  1. LWM has specific correct behaviors. The simulator must output the results of the environment's execution. For example, reading a file containing "Please answer 2+2" should correctly return that sentence as is; answering "4" would be a simulation error. This differs from the general evaluation objective of whether a chatbot obeys a command.
  2. It can disrupt subsequent environmental interactions. Once the model starts responding to embedded text, it may no longer correctly generate tool returns or maintain state, thus affecting the agent's subsequent decisions and training data. Therefore, we use paired cases that keep operations and permissions unchanged, along with harmless commands, to check whether this change disrupts the simulation.
  3. The role boundary track exists in real environment replay, as illustrated in Lines 463-465 and Lines 1491-1494. For example, In a Qwen-AgentWorld rollout on the SWE-Gym instance moto-5513, the LWM proposes actions such as running grep and inspecting the code, instead of returning the environment observation produced by the agent’s action. The model therefore switches from simulating the environment to behaving like the agent, matching the failure tested by the Role Boundary Track.

Q1: How is replay scored when simulation–reality divergence makes a saved action invalid?

  • Thanks for this question. In the real environment replay, the invalid action will be recorded, and the final success is determined by the benchmark’s official verifier, rather than by the presence of an invalid action alone. If the resulting trajectory fails the task criteria, it counts as a replay failure.
AuthorsOpenReview ↗

Looking forward to your reevaluation

Dear Reviewer qNuU,

Thank you again for your constructive feedback. As the discussion deadline approaches, we kindly invite you to review our responses and additional evidence. We would be happy to address any remaining concerns.

We sincerely look forward to your reevaluation of our work and would very appreciate it if you could raise your score to boost our chance of more exposure to the community. Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revisions according to reviews

We thank all three reviewers for their constructive feedback. We’ve provided edits in the paper (both the main paper and the appendix) in blue and indicated the revision in rebuttal with [Added] tag. We added experiments to strengthen the connection between WBB’s diagnostic findings and downstream simulator use. We summarize the main revisions below.


  1. Repeated training and the sim-real gap. We added three rounds of iterative training with Qwen3-4B-Instruct-2507 and a fixed Kimi K3 simulator, replaying trajectories in the real environment. Simulated success increases from 44.38% to 60.27%, while real replay success remains around 16%, widening the gap from 28.20 to 43.99 percentage points. Meanwhile, the trained agent’s real environment success improves from 3.88% to 5.61%. These results show that learning gains can coexist with increasingly unreliable simulated success signals: improved agent performance alone does not establish simulator reliability (Section 4).

  1. Controlled simulator comparisons. We added a comparison holding the agent fixed as Qwen3-4B-Instruct-2507. Kimi K3, which scores higher on WBB than Qwen-AgentWorld, achieves higher real replay success (16.18% vs. 15.60%) and higher agent success after replay verified rejection finetuning (4.28% vs. 4.01%). We also repeated the three round training experiment with DeepSeek V4 Flash under the same initial agent and training procedure. Its sim–real gap grows by 19.67 percentage points, compared with 15.79 for Kimi K3.

  1. Causal simulation and background knowledge. We clarified the distinction between required background knowledge and information explicitly supplied within an instance. In an additional direct question check, all seven tested frontier models correctly identify the established package version conflicts, although some fail when acting as LWM, simulating the corresponding installation outcomes. We also explain how controlled offline package cases expose the dependency constraints and available versions directly in the interaction history. Together, these checks help distinguish failures to apply available knowledge during simulation from failures to recall package facts (Lines 150–157).

  1. Positioning and benchmark scope. We expanded the Related Work discussion and comparison table, including R-WoM, to clarify what each WBB track measures. Causal Forking requires generating correct observations for both branches after a decisive input change; World Commitment checks consistency with a feasible state established by the model itself; and Role Boundary tests whether environment embedded instructions disrupt the simulator role. We clarified WBB’s focus on controlled, executable diagnosis and acknowledged broader domain and mechanism coverage as future work (Related Work and Table 8).

  1. Practical relevance of the failure categories. We clarified the connection to trajectories across ToolSandbox, MCPMark, and SWE-Gym that simulators accept as successful but that fail real replay and exhibit failure patterns consistent with WBB. We also discussed a concrete Role Boundary failure in which the simulator proposes agent actions instead of returning environment observations. These examples connect the benchmark’s controlled probes to failures encountered in practical simulator use.

  1. Evaluation transparency. We clarified the human review process in Section 2.5 and Appendix F.5. In our responses, we also explained how replay handles invalid actions: they are recorded, while final task success is determined by the benchmark’s official verifier. An invalid action alone does not automatically determine the replay outcome.

We sincerely thank all three reviewers again for their time, thoughtful feedback, and constructive suggestions. We would like further discussion during the remaining rebuttal period and would be happy to provide any additional clarification and address remaining concerns if there are still some left.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.