acceptodds
Under review as a conference paper at ICLR 2027

ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

Abstract

Tool-using agents alternate between model generation and tool execution, waiting for each tool call to complete before starting the next generation. This sequential execution is costly during agentic reinforcement learning (RL), which requires large batches of multi-turn rollouts. We introduce ObsSpec, an observation speculation pipeline for agentic RL that uses a small world model to predict tool outputs while the environment executes, allowing the policy to begin its next generation early. ObsSpec retains speculative work only when the predicted and actual observations match exactly, making rollout collection lossless, while co-training keeps the world model up to date as the policy learns. On terminal-use tasks, predictions from a 0.6B world model exactly match actual observations in 66.4% of attempts and arrive roughly 16 times faster. Using these predictions, ObsSpec reduces agent latency by 9.3% and 14.2% for 8B and 32B policies. Across controlled experiments, we characterize the regimes in which ObsSpec reduces agent latency under shared policy resources. Finally, because ObsSpec overlaps policy generation with tool execution, it is architecturally compatible with existing techniques for accelerating RL.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, tool-using agents, observation speculation, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7UjFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a lossless speculative decoding technique to speed up the rollout generation during RL training. It uses a small, co-trained “world model” that predicts tool responses while the actual tool calls are still executing, so generation can continue from the predicted response. To keep rollouts identical to the normal ones, the rollout is kept only if the real tool response matches the prediction exactly when the call finishes. Otherwise, the rollout is rolled back to that point and continues from the actual response. The authors run a series of experiments on terminal-use style tasks to measure the efficiency and latency gains from their technique. They compare RL training runs with and without a world model (Qwen3 0.6B) and report that it reduces end-to-end rollout generation latency by about 14% for Qwen3 32B and about 8% for Qwen3 8B.

Critical strengths

• Because rollouts are exact, better(more accurate) or faster(lower latency) world models can potentially bring more speed. So, better or task-specific world models could bring much larger efficiency gains in settings where tool responses are predictable but slow to actually execute.

Critical weaknesses

• The paper mentions that basic terminal commands such as ls, mkdir, and echo take about 1.6s, which is surprisingly slow. The authors should state more clearly how much of this comes from the harness or containerization, and whether that overhead could be cut substantially with engineering changes. • The comparisons with and without the world model are not compute matched. The world model is trained and served separately on additional GPUs, so it is a bit unclear whether there is a net gain at equal compute.

Questions for authors

• Are there ablations where the RL training run uses the same total number of GPUs, with the world model trained online inside that budget, so the two setups can be compared directly? • How much of the tool call latency comes from the harness, and how much faster could it be made?

2 comments

We thank the reviewer for raising these points on tool latency and compute, and respond to both concerns below.

W1 / Q2. Tool-call latency.

For basic terminal commands such as ls, mkdir, and echo, most of the latency is the round trip rather than the command itself. That round trip includes the network hop to the sandbox, scheduling under high concurrency, shell startup, and saving and restoring the shell state. We will clarify this in the paper.

W2 / Q1. Equal GPU budget.

We dedicated 8 GPUs (of the same type as the policy GPUs) to train and serve the world model in all our 8B and 32B policy experiments. To ensure a fair comparison against the sequential baseline, we fixed the workload and reran the sequential comparison under equivalent GPU budgets.

SetupGPUsAgent (s)65K batch (s)Async (rollouts/h)
Sequential843.2±0.4156.3±2.913,762±156
Sequential1640.0±0.4154.3±3.014,350±160
ObsSpec1638.8±0.4149.4±3.114,571±175

While the 16-GPU sequential baseline improves over the 8-GPU sequential baseline, it falls short of ObsSpec on all three metrics: agent latency, batch latency, and async throughput. Moreover, allocating 8 GPUs to the small world model is indeed overkill, and we can achieve comparable agent latency with just 1 GPU (see the table below). In any case, the world model requires significantly less GPU compute than the policy, and its cost does not scale with policy size: the same 0.6B world model serves both the 8B and 32B policies. Alternatively, the world model could run on lower-performance GPUs. We will incorporate these new results into the paper draft.

WM GPUsAgent (s)
139.3±0.4
239.0±0.4
438.9±0.4
838.8±0.4

I have read all the comments carefully. However, the responses do not address the concerns I raised (the net gains are not high), and I therefore maintain my original score.

Reviewer_FyBLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The ObsSpec framework enhances agentic reinforcement learning by forecasting pending tool outputs, thereby enabling the initiation of subsequent policy generation prior to the completion of tool execution. Speculative computations are preserved solely when the predicted observations precisely align with the actual ones; following verification, subsequent actions proceed in a sequential manner. The proposed pipeline refreshes the world model using ground-truth observations while constraining both speculation depth and breadth to mitigate inference contention. Experimental evaluations employ GRPO-trained Qwen3-8B/32B policies on the Endless Terminals benchmark, where world models are trained offline via sequential replay of stored training trajectories. A 0.6B-parameter world model achieves an approximate 66.4% rate of perfect correspondences. Under fixed policy-serving resources supplemented by additional world-model GPUs, ObsSpec yields reductions in agent latency by 9.3%/14.2%, batch latency by 4.4%/7.8%, and improvements in asynchronous throughput by 5.9%/11.5%, while maintaining comparable mean rewards.

Critical strengths
  1. The pipeline provides a clear and well-motivated safeguard for RL trajectory validity. Exact observation validation, rollback of incorrect branches, and execution of subsequent actions only after validation allow speculative computation to exploit tool waiting time while preventing incorrect predictions and their dependent actions from entering training trajectories.

  2. The measurements and ablations under shared inference resources provide useful systems evidence. The paper reports agent latency, batch latency, and asynchronous throughput, and uses decode rates and discarded tokens to examine contention. For example, increasing breadth from 1 to 8 improves the match rate by only approximately 2.6 percentage points while increasing agent latency by 25.3%.

Critical weaknesses
  1. The lack of a comparison under equal total resource budgets undermines the claimed computational efficiency advantage. In Section 3.1, the 8B and 32B policies are allocated 8 and 16 GPUs respectively, with an additional 8 GPUs dedicated to the world model, whereas the baseline employs only policy-serving GPUs. Consequently, total serving GPU allocation increases by 100% (for the 8B policy) and 50% (for the 32B policy), yielding asynchronous throughput improvements of 5.9% and 11.5%, respectively. The study does not provide an equal-total-GPU sequential baseline, a systematic ablation that reduces the number of world-model replicas, or a comprehensive GPU-hours or cost analysis. While the results demonstrate reduced latency under increased resource provisioning, they fail to establish whether ObsSpec constitutes a more effective use of those resources compared to alternative allocation strategies.

  2. Evidence regarding the acceleration of online RL and the contribution of co-training remains insufficient. Section 3.1 indicates that policy training trajectories are saved and later replayed offline to train the world model. Although this design reasonably controls policy behavior during comparisons of rollout configurations, it does not account for world-model updates, weight synchronization overhead, or stage-waiting delays that would arise in an actual joint online training process. Furthermore, no measurement is provided of the time required to reach a matched reward level. Additionally, the comparison in Section 3.6 involves an updated world model versus one frozen after step 100, thereby confounding differences in both training exposure and data recency.

Questions for authors

1.​ First, are the eight world-model GPUs of the same hardware specification as the policy GPUs, fully dedicated to this workload, and essential for sustaining the reported throughput levels? Second, under a fixed total GPU budget, how does ObsSpec’s throughput and total GPU-hour consumption compare with a sequential rollout baseline that instead receives those same resources?

2.​ When controlling for the number of optimization steps or the total token budget, how do prediction accuracy and achieved latency differ between a world model that is further trained on data from an earlier policy checkpoint and one trained on trajectories from the most recent policy? Such a comparison would help decouple the influence of extended training from that of an updated data distribution.

2 comments

We thank the reviewer for the careful reading of the experiments, and address the questions and concerns below.

W1 / Q1. Equal GPU budget.

We dedicated 8 GPUs (of the same type as the policy GPUs) to train and serve the world model in all our 8B and 32B policy experiments. To ensure a fair comparison against the sequential baseline, we fixed the workload and reran the sequential comparison under equivalent GPU budgets.

SetupGPUsAgent (s)65K batch (s)Async (rollouts/h)
Sequential843.2±0.4156.3±2.913,762±156
Sequential1640.0±0.4154.3±3.014,350±160
ObsSpec1638.8±0.4149.4±3.114,571±175

While the 16-GPU sequential baseline improves over the 8-GPU sequential baseline, it falls short of ObsSpec on all three metrics: agent latency, batch latency, and async throughput. Moreover, allocating 8 GPUs to the small world model is indeed overkill, and we can achieve comparable agent latency with just 1 GPU (see the table below). Alternatively, the world model could run on lower-performance GPUs. In any case, the world model requires significantly less GPU compute than the policy, and its cost does not scale with policy size: the same 0.6B world model serves both the 8B and 32B policies. We will incorporate these new results into the paper draft.

WM GPUsAgent (s)
139.3±0.4
239.0±0.4
438.9±0.4
838.8±0.4

W2. Joint-training overhead.

ObsSpec's joint training procedure adds negligible overhead. Since the world model and policy are trained and served on separate GPUs, weight updates, synchronization, and checkpointing can run in parallel. In fact, because the world model is often significantly smaller than the policy, these operations are much quicker for the world model than the policy. For instance, the 0.6B world model update averaged just 6.4 seconds, while the 8B policy update averaged 25.07 seconds, and the slowest world model update was faster than the fastest policy update. We will clarify this in the paper.

Q2. Data recency vs extra optimization.

Section 3.6 compares a world model frozen after step 100 with one trained through step 500. That comparison mixes extra optimizer steps with more recent data. To separate the two, we conducted additional experiments:

World modelOptimizer stepsMatchAgent (s)
Steps 1–100, 1 epoch10057.4%39.7±0.4
Steps 1–100, 5 epochs50060.9%40.1±0.4
Steps 401–500, 1 epoch10064.6%38.9±0.4
Trained through step 50050066.9%38.8±0.4

Training with extra optimizer steps did not translate to reduced agent latency, while training on data recently collected from the policy did. We will incorporate these results into Section 3.6 to show that the gap in agent latency comes from data recency, not from the extra optimizer steps.

Reviewer_FyBLOpenReview ↗

Response

I now have a reasonable understanding of the computational overhead involved in the calculations and the ablation studies. I thank the authors for their clarification, which has addressed some of my concerns.

Reviewer_b54AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on speeding up the rollout process in agentic RL. The authors propose ObsSpec, which uses a small world model to predict tool outputs so the agent can start generating the next action early. Once the actual output arrives, the system checks whether the prediction is correct. If it matches, the generated action is kept. Otherwise, the agent generates the action again. This ensures the correctness of the training data, which is similar to speculative decoding.

The authors also update the world model alongside the policy and study how different speculation settings affect performance. Experiments show that ObsSpec reduces agent latency by up to 14.2% and improves rollout throughput by up to 11.5%.

Critical strengths
  • The paper directly addresses the rollout latency bottleneck in agentic RL by overlapping tool execution with policy generation. Unlike prior work mainly focused on inference-time speculation, ObsSpec ensures lossless rollout collection and co-trains the world model to adapt to the evolving policy. This makes the approach particularly suitable for agentic RL training.

  • The paper goes beyond demonstrating speedups and carefully studies speculation under shared policy resources. An interesting finding is that increasing speculation depth or breadth can actually hurt performance due to GPU contention. These results provide useful guidance for designing efficient speculation pipelines in large-scale agentic RL systems.

Critical weaknesses
  • Limited comparison with existing acceleration methods. The paper mainly compares ObsSpec against a sequential baseline. Although several related speculation-based methods are discussed, there is no direct experimental comparison with them. Including such comparisons would help better demonstrate the advantages of ObsSpec over existing approaches.
  • Limited evaluation on diverse agent tasks. The experiments mainly focus on terminal-use tasks. Although the paper also includes a TIR workload, evaluating ObsSpec on other agent settings, such as web or API-based agents, would help demonstrate its broader applicability.
Questions for authors
  • The paper shows that co-training helps the world model adapt to policy updates. However, how stable is this process throughout RL training? For example, could rapid changes in the policy distribution cause fluctuations in prediction accuracy or affect the overall speedup?

  • ObsSpec requires additional GPU resources for world model training and inference. How much overhead does this introduce? Would the method still provide an overall training speedup when considering the extra computation and using the same total GPU budget?

1 comment

We thank the reviewer for the constructive comments, and respond to the questions and concerns below.

W1. Comparison with existing methods.

Several existing methods align closely with configurations we studied in Section 3.5, and we will make this connection explicit in the paper. For instance, our breadth experiments match Speculative Actions [1], where increasing the breadth from K=1 to K=4 increased agent latency by 17.2% and reduced the policy decode rate by 19.7%. Likewise, SpecHop [2] aligns with our depth experiments under a looser rule for accepting speculative observations. Under that rule, the match rate increased from 64.7% to 73.0%, but the mean reward fell from 78.3% to 72.1% and agent latency increased from 38.8s to 40.1s.

W2. Task scope.

Effective observation speculation requires predictable observations and state transitions long enough to hide policy generation. Terminal tasks, along with other coding and SWE tasks, meet both requirements, while TIR meets neither. Web and search tasks have observations that are difficult to predict exactly, whereas ALFWorld is predictable but its transitions are too short to hide policy generation. Future work should study looser verification strategies in order to generalize to tasks whose observations are difficult to predict exactly.

Q1. Co-training stability.

Section 3.6 shows that the policy distribution evolves gradually, not rapidly. The difference in match rate between a frozen and an updated world model grew steadily from 0 to 9.1 percentage points over 400 optimizer steps. Even if the world model cannot keep up with the policy, one can use a higher learning rate or more optimizer steps for the world model. Importantly, an inaccurate world model affects training throughput and not correctness. We will clarify this in the manuscript.

Q2. Equal GPU budget.

We dedicated 8 GPUs (of the same type as the policy GPUs) to train and serve the world model in all our 8B and 32B policy experiments. To ensure a fair comparison against the sequential baseline, we fixed the workload and reran the sequential comparison under equivalent GPU budgets.

SetupGPUsAgent (s)65K batch (s)Async (rollouts/h)
Sequential843.2±0.4156.3±2.913,762±156
Sequential1640.0±0.4154.3±3.014,350±160
ObsSpec1638.8±0.4149.4±3.114,571±175

While the 16-GPU sequential baseline improves over the 8-GPU sequential baseline, it falls short of ObsSpec on all three metrics: agent latency, batch latency, and async throughput. Moreover, allocating 8 GPUs to the small world model is indeed overkill, and we can achieve comparable agent latency with just 1 GPU (see the table below). Alternatively, the world model could run on lower-performance GPUs. In any case, the world model requires significantly less GPU compute than the policy, and its cost does not scale with policy size: the same 0.6B world model serves both the 8B and 32B policies. We will incorporate these new results into the paper draft.

WM GPUsAgent (s)
139.3±0.4
239.0±0.4
438.9±0.4
838.8±0.4

[1] Ye, Naimeng, et al. "Speculative actions: A lossless framework for faster AI agents." The Fourteenth International Conference on Learning Representations. 2026.

[2] Saberi, Mehrdad, et al. "SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents." arXiv preprint arXiv:2605.21965. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.