- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposed EchoRL, a runtime layer that reuses cached execution results during agentic RL training. The paper makes two contributions: reduce environment latency by casting reuse as caching a deterministic transition function keyed by logical state and tool version, and show that under an explicit determinism assumption cached and uncached rollouts follow the same distribution. The results show that EchoRL reduces per-rollout environment latency.
Critical strengths
- The motivation and research problem is interesting and practical.
- The proposed method is intuitive and should be able to easily adapt to different model and tasks.
Critical weaknesses
- The biggest concern I have is the assumption that same logical state, action, tool will generate the exact same result. The entire paper is built upon and motivated by this assumption. However, I didn't find the relevant empirical experiment results on this. In real coding agent environment, this assumption might not be valid since there any many other dependencies, such as external API, packages, etc. We need experiments results to prove this assumption is valid.
- The conclusion will be more conniving if we can extend the full RL training evaluation to other models and other family. The current result only evaluate one model for full RL training, and lack of cross-family eval, not sure if this is a general approach for all models.
Questions for authors
The proposed advantage-guided retention didn't be ablated to evaluate only by itself. It's unclear how much improvement we can get from this module.
3 comments
Thanks author for the detailed response. Most of my concerns are addressed.
For "It hashes the initial environment and the exact sequence of mutations, so a mismatch can only cause a missed hit, not an incorrect one." I wonder is this true? This should only works if all relevant mutations are captured and the initial environment descriptor includes all relevant state. For example, untracked filesystem changes, environment variables, or externally modified resources could cause logically different states to share a cache key.
I will change my score accordingly if author can well-justify this question.
Thank you for the follow-up. EchoRL enforces both conditions in the runtime with a default-deny rule. State is tracked only through actions whose effect is known, and any action whose effect is not known turns reuse off for the rest of that rollout instead of being ignored. We explain how this works for each of the three examples.
1. Untracked filesystem changes. Two properties rule these out.
- Closed sandbox. Each rollout runs in its own private environment. Restored instances are private and copy-on-write, and cached snapshots are immutable (Appendix B.4). Other rollouts and the host therefore cannot modify a rollout's filesystem. The only source of mutation is the rollout's own tool calls, and all of them pass through the EchoRL executor.
- Default-deny classification. Every tool call falls into one of three classes (Appendix B.3). A verified read-only action leaves the logical state unchanged. A verified deterministic state-changing action advances the logical state by hashing the action into it. Every other action, including any command outside the whitelist or with an unknown effect, executes normally and then disables reuse for all later steps of that rollout. For example, in Appendix C.2.3 the command
sort -u /app/input.txt > /app/output.txtis outside the whitelist because of the redirection, so it runs normally and all subsequent commands in that rollout bypass the cache.
A mutation is therefore either hashed into the key or it ends reuse. There is no path by which a mutation is silently dropped while the rollout continues to hit the cache.
2. Environment variables. These are part of the key. The transition descriptor is q_t = Serialize(name, args, cwd, env, v_t) (Eq. 26), so the same command under a different working directory or environment produces a different key. In addition, in Terminal-Bench each command runs in a new bash shell in the initial working directory (Appendix C.2.3), so an export in one command does not carry over to the next. A persistent change, such as editing a shell profile, is a filesystem mutation and is handled as in point 1.
3. Externally modified resources.
- Initial state. The initial descriptor hashes the task identity, the digest of the base image, the initialization procedure, and the task configuration (Eq. 25). The image digest is a content hash, so every file in the image is covered. The same holds for task data: SkyRL-SQL keys on the database digest and EgoSchema keys on the SHA-256 of the video (Appendix B.2). If any of these changes, the key changes.
- State outside the sandbox. Mutable remote services, unversioned mounted files, system time, and unrecorded randomness are non-cacheable unless they are captured in q_t or v_t, and an action that depends on them also disables reuse of descendant states (Appendix A.1). Network-dependent steps are eligible only with pinned inputs, such as a dependency installation with pinned versions.
Additional safeguards. The whitelist of read-only and deterministic commands is kept narrow and conservative, so a command is treated as cacheable only when its effect is known. On an L2 hit, EchoRL also verifies the successor logical-state descriptor of the restored environment before it returns the cached observation (Appendix B.4). Finally, the implementation provides a shadow audit that re-executes an action on a fork of the pre-action state and compares both the observation and the resulting state fingerprint. A mismatch invalidates the entry (Appendix B.3).
In summary, with sandbox isolation and default-deny action classification, two different logical states do not share a cache key, and a descriptor mismatch can only cause a missed hit. We will move this explanation from the appendix into Sec. 3.2 so that it is clear in the main text.
We thank the reviewer for the careful reading and constructive comments.
W1: Validity of the determinism assumption
We agree that this does not hold for arbitrary tool calls in real coding environments. EchoRL does not assume it globally. Assumption 1 is a per-transition eligibility condition that the runtime enforces, and anything that fails it is executed normally. We address the concern from two angles.
(1) Only pre-screened deterministic operations are ever cached
Before training, we screen the tool set of each workload and mark as cacheable only operations whose outputs are fixed given their inputs, such as file reads and inspections and fixed processing steps over pinned inputs. Everything else bypasses the cache.
EXPLAIN. Queries that use functions outside a deterministic whitelist bypass reuse (Appendix B.2).The dependencies the reviewer mentions are handled explicitly:
The logical-state descriptor is also deliberately conservative. It hashes the initial environment and the exact sequence of mutations, so a mismatch can only cause a missed hit, not an incorrect one. The cost of non-determinism in real environments is therefore a lower hit rate, not a wrong trajectory.
(2) Training behavior is unchanged with and without EchoRL
If cached transitions deviated from real execution, the policy would be trained on corrupted observations and its learning curve would diverge from uncached training. Figure 3(a–b) shows that held-out test accuracy with and without EchoRL remains comparable at matched training iterations on EgoSchema and SkyRL-SQL, with no systematic degradation. Held-out evaluation is run with environment reuse disabled (Appendix B.5), so these accuracies are measured on real executions. This is end-to-end evidence that the assumption holds for the operations EchoRL actually caches in these workloads.
We will move the screening procedure and the non-cacheable rules from the appendix into Sec. 3.2 so that the scope of Assumption 1 is clear in the main text.
W2: Full RL training on other models and model families
Appendix D.2 (Table 3) reports results beyond the Qwen3.5-9B model used in the main experiments. On Terminal-Bench, EchoRL consistently reduces per-rollout environment execution time across a roughly 7× range of model sizes:
We acknowledge that these models belong to one family. EchoRL is model-agnostic by construction. It sits between the inference engine and the environment backend, and it observes only tool invocations and scalar advantages. It never accesses model weights, tokenizers, or architecture-specific state. The policy model affects EchoRL only through the trajectories it generates, which changes how much reuse is available but not whether reuse is correct. We will state this limitation more explicitly in the revision.
Q: Advantage-guided retention (AGR) is not ablated by itself
AGR is a retention and eviction policy over L2/L3 cache entries, so it cannot run without the cache hierarchy. The meaningful isolation is the same hierarchy with and without AGR, where the variant without AGR uses plain LRU. We report exactly this comparison on Terminal-Bench.
The gain in (c) therefore comes from the higher hit rate in (d): AGR keeps the states that the updated policy is more likely to revisit. We will label the "w/o AGR" variant more clearly in the revision.