- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a lossless speculative decoding technique to speed up the rollout generation during RL training. It uses a small, co-trained “world model” that predicts tool responses while the actual tool calls are still executing, so generation can continue from the predicted response. To keep rollouts identical to the normal ones, the rollout is kept only if the real tool response matches the prediction exactly when the call finishes. Otherwise, the rollout is rolled back to that point and continues from the actual response. The authors run a series of experiments on terminal-use style tasks to measure the efficiency and latency gains from their technique. They compare RL training runs with and without a world model (Qwen3 0.6B) and report that it reduces end-to-end rollout generation latency by about 14% for Qwen3 32B and about 8% for Qwen3 8B.
Critical strengths
• Because rollouts are exact, better(more accurate) or faster(lower latency) world models can potentially bring more speed. So, better or task-specific world models could bring much larger efficiency gains in settings where tool responses are predictable but slow to actually execute.
Critical weaknesses
• The paper mentions that basic terminal commands such as ls, mkdir, and echo take about 1.6s, which is surprisingly slow. The authors should state more clearly how much of this comes from the harness or containerization, and whether that overhead could be cut substantially with engineering changes. • The comparisons with and without the world model are not compute matched. The world model is trained and served separately on additional GPUs, so it is a bit unclear whether there is a net gain at equal compute.
Questions for authors
• Are there ablations where the RL training run uses the same total number of GPUs, with the world model trained online inside that budget, so the two setups can be compared directly? • How much of the tool call latency comes from the harness, and how much faster could it be made?
2 comments
I have read all the comments carefully. However, the responses do not address the concerns I raised (the net gains are not high), and I therefore maintain my original score.
We thank the reviewer for raising these points on tool latency and compute, and respond to both concerns below.
W1 / Q2. Tool-call latency.
For basic terminal commands such as ls, mkdir, and echo, most of the latency is the round trip rather than the command itself. That round trip includes the network hop to the sandbox, scheduling under high concurrency, shell startup, and saving and restoring the shell state. We will clarify this in the paper.
W2 / Q1. Equal GPU budget.
We dedicated 8 GPUs (of the same type as the policy GPUs) to train and serve the world model in all our 8B and 32B policy experiments. To ensure a fair comparison against the sequential baseline, we fixed the workload and reran the sequential comparison under equivalent GPU budgets.
While the 16-GPU sequential baseline improves over the 8-GPU sequential baseline, it falls short of ObsSpec on all three metrics: agent latency, batch latency, and async throughput. Moreover, allocating 8 GPUs to the small world model is indeed overkill, and we can achieve comparable agent latency with just 1 GPU (see the table below). In any case, the world model requires significantly less GPU compute than the policy, and its cost does not scale with policy size: the same 0.6B world model serves both the 8B and 32B policies. Alternatively, the world model could run on lower-performance GPUs. We will incorporate these new results into the paper draft.