- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adds an online transition table, episodic retrieval, and a confusion tracker to an LLM-agent speculator, improving action, observation, and chained predictions while the actor or environment is busy. Six benchmarks compare progressive memory settings with adapted AWM and ReasoningBank baselines. Structured observation tasks show substantial gains: ALFWorld improves from 16.3% to 40.0%, while web-action gains are smaller. The paper also reports live latency traces and estimates potential execution savings. Its main contribution is an empirical study of speculator memory, alongside a proposed execution framework that reuses predictions only after verification.
Critical strengths
-
The method lets the speculator learn from past trajectories while keeping the actor’s decision context unchanged. Transition statistics, episodic retrieval, and confusion tracking capture recurring workflows and prediction errors without modifying the actor.
-
Across six benchmarks covering action, observation, and chained prediction, memory consistently improves prediction accuracy, especially in ALFWorld and PDDL. The ablations are informative: speculation-miss episodes drive much of the observation-prediction gain, while episodic retrieval outperforms the full configuration on VisualWebArena.
Critical weaknesses
-
The acceleration evidence is modest and relies substantially on estimates. HotpotQA reports a 1.05× analytical speedup, while ALFWorld’s estimated savings assume environment delays despite near-zero feedback latency in its text simulator. Live actor/speculator timings are useful but do not establish integrated end-to-end gains. Substantial accuracy improvements therefore offer limited support for broader acceleration claims. Paired online comparisons should include actual delays and retrieval, verification, and scheduling overhead.
-
The lossless guarantee needs clearer assumptions and validation. Exact action matching checks the operation and parameters, and a read-only whitelist limits side effects. Neither ensures that an early read returns the same observation at the actor’s original execution time. In changing environments, cached observations could alter later decisions despite an exact action match. The paper should specify and enforce observation-equivalence conditions and test changes between speculation and verification. This concern limits the general guarantee; it does not establish failures in the reported experiments.
Questions for authors
Can you report a paired end-to-end comparison of non-speculative, stateless speculative, and memory-based speculative execution on one benchmark with actual environment latency?
3 comments
Thanks for the additional results. My concerns are mostly solved. I raised my rating to weak accept
Thank you for the update, and we appreciate the careful review.
Response to Reviewer W8eo
Thank you for identifying the distinction between prediction accuracy, integrated execution time and the conditions required for lossless reuse.
1. Online execution with actual environment latency
Following your suggestion, we evaluated non-speculative execution, stateless speculation and memory-based speculation on the same 80 tasks in each of WebArena's Reddit and Shopping domains, with actual browser execution, no injected delays, and averaged over three runs. We used a Qwen3-235B-A22B actor and Qwen3-32B speculator served through Bedrock, with k=3 for both speculative methods.
These Type 1 experiments pre-execute only eligible read-only navigations. Memory uses Episodic+Miss and updates after each completed task, starting from 20 prior tasks in both domains and using full-page pre-execution.
Memory's observed mean episode time was 5.5% lower than Stateless and 14.9% lower than None on Reddit. On Shopping, the differences were 6.1% versus Stateless and 22.4% versus None. Reported episode time includes retrieval, verification, scheduling and memory updates, but excludes setup, evaluation and cleanup. API cost excludes warm-start collection and infrastructure.
Realized savings depend on both prediction hit rate and the available overlap window (Section 2.1). We use WebArena for integrated browser execution; benchmarks such as ALFWorld and PDDL have fast simulated feedback with little environment waiting time. WebArena also showed a comparatively modest prediction gain in the original replay study (19.8% to 23.7%, +3.9 percentage points; Table 2). To examine broader timing conditions, Section 4.3 and Appendix A.9 combine measured model latencies with assumed environment delays, illustrating how sufficiently early predictions and longer overlap windows can hide more latency per correct prediction.
2. Observation equivalence and lossless reuse
Thank you for raising this concern. Section 2.1 describes the read-only whitelist and exact action-and-argument matching, while Appendix B.4 notes related risks from changing temporal context. We will formalize observation equivalence (state invariance of the read-only target within the short speculation window) as an explicit precondition for lossless reuse.
We conducted additional checks. On PDDL, a scripted Type-2 prototype comparing complete next-actor inputs rejected reuse in all 20 tested cases with relevant state changes. In an earlier HotpotQA online run, 240 of 242 adopted Search observations (99.2%) were identical when re-queried after task completion. The first is controlled validation of a gate; the second is evidence of observed stability, not an adoption-time equivalence guarantee.
In the revision, we will state this observation-equivalence precondition explicitly, specify how volatile or state-dependent endpoints are excluded from the read-only whitelist, and expand our limitations discussion for rapidly changing environments.