- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces World Boundary Bench to evaluate language models as environment simulators. Its three tracks test sensitivity to decisive input changes, state consistency across interactions, and resistance to instruction-like environment data. It evaluates 13 models and includes real-environment replay and exploratory agent fine-tuning experiments.
Critical strengths
The controlled pairs and executable checks provide a concrete way to diagnose simulation errors. Allowing models to establish an initial state and then checking subsequent consistency is a useful evaluation design. The real-environment replay also provides direct evidence that simulated success can be misleading.
Critical weaknesses
-
The additional insight from the benchmark is limited. The main findings—models overlook decisive details, lose state consistency, and follow instructions embedded in data—are plausible failure modes. Quantifying them is useful, but the paper does not sufficiently establish what these measurements tell us about choosing or improving a simulator. In particular, it does not show whether WBB scores predict real-replay success or downstream training benefit.
-
The training experiments do not resolve the reliability concerns motivating the paper. They show small improvements from a fixed set of selected trajectories on ToolSandbox, but do not examine whether continued optimization encourages agents to exploit simulator errors. The Role Boundary track tests fixed instruction-like inputs, rather than agents learning to obtain apparent success through simulator weaknesses. The two downstream configurations also change both the agent and simulator, preventing a controlled assessment of how simulator quality affects learning.
Questions for authors
-
Do WBB scores, or particular error categories, predict downstream replay success or training gains when the agent is held fixed?
-
Does repeated training against a simulator increase the gap between simulated and real success? Can the proposed benchmark detect simulators that are particularly vulnerable to such exploitation?
6 comments
Rebuttal by Authors (2/2)
W2(c): Changing both the agent and simulator confounds the effect of simulator quality on learning.
- Thanks for this question. We complement an new ablation study that using the same agent: Qwen3-4B-Instruct-2507 and Kimi K3, which achieves higher WBB scores, as a comparison to the (Qwen3-4B-Instruct-2507, Qwen-AgentWorld) old pair. The results are shown as following:
| Simulator | Real replay success | Replay verified RFT success |
|---|---|---|
| Qwen-AgentWorld | 15.60% | 4.01% |
| Kimi K3 | 16.18% | 4.28% |
This comparison removes the change in agent and directly evaluates whether trajectories generated through interaction with each simulator succeed under real execution.
Q2(b): Can the proposed benchmark detect simulators that are particularly vulnerable to the enlarging gap between sim and real? [Added]
- Thanks for this question. To investigate this, we additionally conducted three rounds of training with DeepSeek V4 Flash, keeping the initial agent and training procedure consistent with the Kimi K3 experiment.
| Simulator | Round 1 gap | Round 3 gap | Gap increase |
|---|---|---|---|
| Kimi K3 | 28.20 % | 43.99 % | +15.79 % |
| DeepSeek V4 Flash | 22.97 % | 42.64 % | +19.67 % |
Both simulators showed a widening gap between the simulation success rate and the actual playback success rate during three rounds of training, indicating that this phenomenon also occurred in another set of simulator configurations. We have revised the main context and complemented this ablation study in Lines 492-504.
Response to the authors' rebuttal
We thank the authors for a substantive and well-executed response. The three-round iterative training experiment (W2(a)) directly addresses our original question and is well designed: the finding that simulated success rises while real-replay success stays flat, with the gap widening from 28.20 to 43.99 percentage points, is an interesting and carefully demonstrated result — learning gains can coexist with increasingly unreliable simulated success signals. We acknowledge the value of this addition.
However, the additional experiments offered to connect WBB scores to simulator quality raise new concerns rather than resolving our first weakness (W1).
The controlled simulator comparison is within noise. With the agent fixed (Qwen3-4B-Instruct-2507), the difference between Kimi K3 and Qwen-AgentWorld is 0.58 percentage points in real-replay success (16.18% vs. 15.60%) and 0.27 points in replay-verified RFT (4.28% vs. 4.01%). The latter is comparable to the paper's own reported RFT seed standard deviations (±0.13–0.39 across three seeds). No variance or significance is reported for any of these comparisons. Two simulators that differ enormously in WBB scores — Kimi K3 is the strongest overall model evaluated, while Qwen-AgentWorld is among the weakest open-source models — produce downstream benefit differences that are statistically indistinguishable from zero. This is the cleanest available test of the predictive value of WBB scores, and the outcome does not support it.
The gap-growth comparison is metric-dependent and based on n = 2. In Table 2, the conclusion rests on the increase in the gap (+15.79 for Kimi K3 vs. +19.67 for DeepSeek V4 Flash) across two simulators. Yet at round 1 the ordering is reversed: the higher-WBB simulator (Kimi K3, 28.20%) exhibits a larger initial gap than the lower-WBB simulator (DeepSeek V4 Flash, 22.97%), and the final gaps at round 3 are nearly identical (43.99% vs. 42.64%). With two data points, and a conclusion that depends on whether one examines the initial gap, the final gap, or the growth, this comparison cannot support the claim that WBB scores predict exploitation vulnerability. A reasonable confound also remains: the two simulators differ in every dimension, so any difference in gaps cannot be attributed to the boundary performance that WBB measures.
Our position. Our original first weakness was that the paper does not establish what WBB measurements tell us about choosing or improving a simulator. The evidence offered in the rebuttal — a qualitative audit of 488 trajectories, a statistically indistinguishable difference in the controlled comparison, and a metric-dependent n = 2 comparison — does not change this assessment. We note that the paper's most robust practical finding, that self-reported simulated success severely overestimates real-replay validity, follows from the replay-verification protocol itself and remains accessible without WBB. Our assessment is therefore that the paper contains a carefully executed and useful negative result about simulator reliability, but its central contribution — that the benchmark scores carry diagnostic and predictive value — remains unestablished, with the strongest controlled evidence currently pointing against it. Our rating is unchanged.
Rebuttal by Authors (1/2)
Thanks for your valuable review and suggestions. Below we respond to Weaknesses (W) and Questions (Q). We use
[Added]to indicate we add and update the corresponding content about the question or weakness in the main revised paper with blue color.W1&Q1: The benchmark's additional insight and value for choosing or improving simulators are insufficiently established. It’s unclear whether WBB scores predict real-replay success or downstream training benefit.
Thanks for this important question. First, WBB gives insights on where simulators fail and whether training improves these specific capabilities and is a benchmark used to explore the failure boundaries of a simulator through carefully constructed controlled tests. The test conditions and their distribution emphasized in WBB are not the same as those for a specific downstream task. Therefore, it cannot be directly expected that the overall WBB score will rank consistently with the success rate or training gains of a particular downstream task.
For example, Qwen-AgentWorld improves Causal Forking joint accuracy over its base model from 22.0% to 27.0%, while World Commitment remains at 54.0%. Thus, better causal prediction does not automatically mean better state consistency. These separate measurements help users compare simulators for their intended tasks and identify weaknesses to target in further training.
W2(a)&Q2(a): Does repeated optimization encourage and widen the simulated real success gap?
[Added]This experiment reveals a clear distinction between simulated success and actual learning progress. Across three rounds, simulated success rises from 44.38% to 60.27%, while real-replay success remains around 16%. Meanwhile, real-environment performance after SFT increases from 3.88% to 5.61%.
These additional results extend our investigation of the two questions in Section 4. First, regarding “Is success in a simulated world real?”, self-reported success increases substantially while real replay success remains approximately unchanged, widening the gap from 28.20 to 43.99 percentage points. Thus, repeated optimization does not necessarily improve the real execution validity of simulated trajectories, even as their apparent success increases. Second, regarding whether LWM self-selected trajectories lead to serious degradation, we observe no degradation over the three training rounds: despite including trajectories that fail real replay, the self selected training data yields checkpoints whose real environment success increases from 3.88% to 5.61%. Together, these findings show that, in this setting, learning gains from self selected simulation data can coexist with an increasing simulation reality gap.
We have complemented this new experiment and content in our main context in Lines 467-491.
W2(b): The Role Boundary track tests fixed instruction-like inputs, rather than agents learning to obtain apparent success through simulator weaknesses.