acceptodds
Under review as a conference paper at ICLR 2027

EchoRL: Policy-Evolving Environment-State Caches for Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) trains policies that act inside real software environments, such as checking out repositories, installing dependencies, running builds, and invoking tools. Group-based learning algorithms, such as GRPO, sample multiple rollouts per task, causing the same deterministic environment operations to be repeated across rollouts and thereby incurring a considerable fraction of the end-to-end rollout latency. Caching environment operations is a natural solution, but policy evolution across training steps raises two challenges: (i) cached execution results may become invalid under updated context, producing incorrect trajectories and biased policy gradients; and (ii) the cache must continually adapt to retain the environment states most likely to be revisited by future policies. To this end, we present EchoRL, a runtime layer that reuses environment work. We make two contributions. First, we reduce environment latency by casting reuse as caching a deterministic transition function keyed by logical state and tool version, and show that under an explicit determinism assumption cached and uncached rollouts follow the same distribution, introducing no additional bias into the underlying policy-gradient estimator. Second, we introduce advantage-guided cache retention to anticipate environment-state reuse under evolving policies, improving cache hit rates in subsequent training iterations. Across all evaluated workloads, EchoRL significantly reduces per-rollout environment latency by – compared with No Cache while preserving trajectory correctness.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
agentic reinforcement learning, environment-state caches

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fUraOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed EchoRL, a runtime layer that reuses cached execution results during agentic RL training. The paper makes two contributions: reduce environment latency by casting reuse as caching a deterministic transition function keyed by logical state and tool version, and show that under an explicit determinism assumption cached and uncached rollouts follow the same distribution. The results show that EchoRL reduces per-rollout environment latency.

Critical strengths
  • The motivation and research problem is interesting and practical.
  • The proposed method is intuitive and should be able to easily adapt to different model and tasks.
Critical weaknesses
  • The biggest concern I have is the assumption that same logical state, action, tool will generate the exact same result. The entire paper is built upon and motivated by this assumption. However, I didn't find the relevant empirical experiment results on this. In real coding agent environment, this assumption might not be valid since there any many other dependencies, such as external API, packages, etc. We need experiments results to prove this assumption is valid.
  • The conclusion will be more conniving if we can extend the full RL training evaluation to other models and other family. The current result only evaluate one model for full RL training, and lack of cross-family eval, not sure if this is a general approach for all models.
Questions for authors

The proposed advantage-guided retention didn't be ablated to evaluate only by itself. It's unclear how much improvement we can get from this module.

3 comments

We thank the reviewer for the careful reading and constructive comments.

W1: Validity of the determinism assumption

We agree that this does not hold for arbitrary tool calls in real coding environments. EchoRL does not assume it globally. Assumption 1 is a per-transition eligibility condition that the runtime enforces, and anything that fails it is executed normally. We address the concern from two angles.

(1) Only pre-screened deterministic operations are ever cached

Before training, we screen the tool set of each workload and mark as cacheable only operations whose outputs are fixed given their inputs, such as file reads and inspections and fixed processing steps over pinned inputs. Everything else bypasses the cache.

  • SkyRL-SQL: databases are opened read-only and immutable, and each query is classified via EXPLAIN. Queries that use functions outside a deterministic whitelist bypass reuse (Appendix B.2).
  • EgoSchema: each video is identified by its SHA-256 digest. The video tools are served by a frozen model at temperature 0 with a fixed seed and deterministic inference enabled (Appendix B.2, C.1.2).
  • Terminal-Bench: shell commands are classified by a command policy. Only whitelisted deterministic commands are eligible, for example dependency installation with pinned inputs or reproducible compilation (Appendix B.2).

The dependencies the reviewer mentions are handled explicitly:

  • Packages and toolchains are part of the cache key through the version descriptor v. For a FlashAttention source build, this covers the source commit, Python, PyTorch, CUDA, compiler, and build configuration (Appendix A.1). A change in any of them yields a different key, so a stale result is never returned.
  • External APIs, mutable remote services, system time, and unrecorded randomness are marked non-cacheable. Once such an action mutates the environment, EchoRL also disables reuse for all descendant states of that rollout (Sec. 3.2, Appendix A.1).

The logical-state descriptor is also deliberately conservative. It hashes the initial environment and the exact sequence of mutations, so a mismatch can only cause a missed hit, not an incorrect one. The cost of non-determinism in real environments is therefore a lower hit rate, not a wrong trajectory.

(2) Training behavior is unchanged with and without EchoRL

If cached transitions deviated from real execution, the policy would be trained on corrupted observations and its learning curve would diverge from uncached training. Figure 3(a–b) shows that held-out test accuracy with and without EchoRL remains comparable at matched training iterations on EgoSchema and SkyRL-SQL, with no systematic degradation. Held-out evaluation is run with environment reuse disabled (Appendix B.5), so these accuracies are measured on real executions. This is end-to-end evidence that the assumption holds for the operations EchoRL actually caches in these workloads.

We will move the screening procedure and the non-cacheable rules from the appendix into Sec. 3.2 so that the scope of Assumption 1 is clear in the main text.

W2: Full RL training on other models and model families

Appendix D.2 (Table 3) reports results beyond the Qwen3.5-9B model used in the main experiments. On Terminal-Bench, EchoRL consistently reduces per-rollout environment execution time across a roughly 7× range of model sizes:

ModelNo Cache (s/rollout)EchoRL (s/rollout)Speedup
Qwen3.5-4B184.633.15.6×
Qwen3.5-9B209.835.75.9×
Qwen3.5-27B317.247.36.7×

We acknowledge that these models belong to one family. EchoRL is model-agnostic by construction. It sits between the inference engine and the environment backend, and it observes only tool invocations and scalar advantages. It never accesses model weights, tokenizers, or architecture-specific state. The policy model affects EchoRL only through the trajectories it generates, which changes how much reuse is available but not whether reuse is correct. We will state this limitation more explicitly in the revision.

Q: Advantage-guided retention (AGR) is not ablated by itself

AGR is a retention and eviction policy over L2/L3 cache entries, so it cannot run without the cache hierarchy. The meaningful isolation is the same hierarchy with and without AGR, where the variant without AGR uses plain LRU. We report exactly this comparison on Terminal-Bench.

  • Figure 3(c), speedup: the full hierarchy without AGR achieves a 5.4× tool-call speedup, and enabling AGR raises it to 6.2×. This is about a 13% further reduction in tool-call latency attributable to AGR alone.
  • Figure 3(d), mechanism: AGR generally maintains a higher cache hit rate across training epochs.

The gain in (c) therefore comes from the higher hit rate in (d): AGR keeps the states that the updated policy is more likely to revisit. We will label the "w/o AGR" variant more clearly in the revision.

Thanks author for the detailed response. Most of my concerns are addressed.

For "It hashes the initial environment and the exact sequence of mutations, so a mismatch can only cause a missed hit, not an incorrect one." I wonder is this true? This should only works if all relevant mutations are captured and the initial environment descriptor includes all relevant state. For example, untracked filesystem changes, environment variables, or externally modified resources could cause logically different states to share a cache key.

I will change my score accordingly if author can well-justify this question.

Thank you for the follow-up. EchoRL enforces both conditions in the runtime with a default-deny rule. State is tracked only through actions whose effect is known, and any action whose effect is not known turns reuse off for the rest of that rollout instead of being ignored. We explain how this works for each of the three examples.

1. Untracked filesystem changes. Two properties rule these out.

  • Closed sandbox. Each rollout runs in its own private environment. Restored instances are private and copy-on-write, and cached snapshots are immutable (Appendix B.4). Other rollouts and the host therefore cannot modify a rollout's filesystem. The only source of mutation is the rollout's own tool calls, and all of them pass through the EchoRL executor.
  • Default-deny classification. Every tool call falls into one of three classes (Appendix B.3). A verified read-only action leaves the logical state unchanged. A verified deterministic state-changing action advances the logical state by hashing the action into it. Every other action, including any command outside the whitelist or with an unknown effect, executes normally and then disables reuse for all later steps of that rollout. For example, in Appendix C.2.3 the command sort -u /app/input.txt > /app/output.txt is outside the whitelist because of the redirection, so it runs normally and all subsequent commands in that rollout bypass the cache.

A mutation is therefore either hashed into the key or it ends reuse. There is no path by which a mutation is silently dropped while the rollout continues to hit the cache.

2. Environment variables. These are part of the key. The transition descriptor is q_t = Serialize(name, args, cwd, env, v_t) (Eq. 26), so the same command under a different working directory or environment produces a different key. In addition, in Terminal-Bench each command runs in a new bash shell in the initial working directory (Appendix C.2.3), so an export in one command does not carry over to the next. A persistent change, such as editing a shell profile, is a filesystem mutation and is handled as in point 1.

3. Externally modified resources.

  • Initial state. The initial descriptor hashes the task identity, the digest of the base image, the initialization procedure, and the task configuration (Eq. 25). The image digest is a content hash, so every file in the image is covered. The same holds for task data: SkyRL-SQL keys on the database digest and EgoSchema keys on the SHA-256 of the video (Appendix B.2). If any of these changes, the key changes.
  • State outside the sandbox. Mutable remote services, unversioned mounted files, system time, and unrecorded randomness are non-cacheable unless they are captured in q_t or v_t, and an action that depends on them also disables reuse of descendant states (Appendix A.1). Network-dependent steps are eligible only with pinned inputs, such as a dependency installation with pinned versions.

Additional safeguards. The whitelist of read-only and deterministic commands is kept narrow and conservative, so a command is treated as cacheable only when its effect is known. On an L2 hit, EchoRL also verifies the successor logical-state descriptor of the restored environment before it returns the cached observation (Appendix B.4). Finally, the implementation provides a shadow audit that re-executes an action on a fork of the pre-action state and compares both the observation and the resulting state fingerprint. A mismatch invalidates the entry (Appendix B.3).

In summary, with sandbox isolation and default-deny action classification, two different logical states do not share a cache key, and a descriptor mismatch can only cause a missed hit. We will move this explanation from the appendix into Sec. 3.2 so that it is clear in the main text.

Reviewer_CXjHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EchoRL caches task initialization, intermediate environment states, and deterministic tool outputs during agentic RL. It combines cost-based admission with an advantage-guided retention rule and gives a conditional argument that transparent cache hits preserve rollout distributions. Experiments on Terminal-Bench, EgoSchema, and SkyRL-SQL report 2.1–8.3× reductions in median environment execution time, with an additional benefit from retention guidance on Terminal-Bench.

Critical strengths
  1. The distinction between reusing an observation and restoring its associated environment state is essential for stateful agents. The design addresses this explicitly through mutation-history identifiers, private restoration, and fresh reward computation. This is a useful systems contribution beyond treating tool calls as ordinary function memoization.

  2. The evaluation covers substantially different environment costs, and the hierarchy ablation makes the contribution of intermediate-state reuse visible: adding L2 raises Terminal-Bench tool-call speedup from 1.5× to 4.3×. The consistent improvement over TVCache is also more informative than the comparison with an uncached system alone.

Critical weaknesses
  1. The correctness argument has not been connected convincingly to the evaluated terminal runtime. Appendix C.2.3 bounds tool timeouts by the remaining per-task agent budget. Skipping earlier execution can therefore change whether a later command times out, even when its filesystem inputs and requested command are identical. Matching mutation-history descriptors does not resolve this difference. Theorem 1 needs an explicit treatment of this execution-budget state, or a restriction excluding such effects. The reported learning-quality comparisons cover only EgoSchema and SQL, while the optional shadow audit defaults to zero (Appendix B.3); neither establishes trajectory preservation for the workload with the most consequential state restoration.

  2. AGR can become stuck on obsolete states. Unvisited entries retain their advantage scores, while new, unscored entries cannot displace positive-score residents (§3.3; Appendix B.5). A cache filled with formerly positive entries can thus block newly relevant states after the policy changes. Expiration is mentioned but not specified or evaluated. The approximately three-point final hit-rate improvement leaves this central adaptation concern unresolved without evidence about stale occupancy and admission failures under capacity pressure.

Questions for authors
  1. How are remaining task time and timeout outcomes handled on cache hits? Can you provide a paired Terminal-Bench audit comparing observations, termination, and verifier outcomes under the actual budget rules, including trajectories that approach their deadlines?

  2. Can a candidate rejected because positive-score entries occupy the cache receive advantage feedback and later displace them? Please specify the expiration policy and show whether stale positive entries block admission under the cache budgets used in Figure 3(d).

  3. What is the end-to-end training wall time at matched updates and task coverage, including inference, optimization, snapshot creation, and waiting? Table 1 measures environment time alone; the 8.3× SQL result saves approximately 0.10 seconds per rollout, so its practical training benefit is currently unclear.

3 comments

We thank the reviewer for the close reading. Both weaknesses point to real gaps in the submission, and we address them directly.

W1 and Q1: Execution-budget state, Theorem 1, and remaining task time on cache hits

The reviewer's observation is correct. The Terminal-Bench agent budget is measured in wall-clock time. On a cache hit, the skipped execution time is not charged to this budget, so a cached rollout has more remaining time than an uncached one at the same step. The logical-state descriptor records the mutation history and does not include the remaining budget, so it cannot remove this difference.

We clarify the scope of Theorem 1 as follows.

  • Transition level. The result holds as stated. For every cache hit, the returned observation and successor logical state equal those of direct execution of the same command to completion.
  • Trajectory level. Eq. (21) additionally requires that termination and timeout outcomes do not depend on wall-clock time. This holds for the turn limit and the token limits. It does not hold when the wall-clock budget is binding.
  • Direction and extent of the deviation. The deviation is one-directional. Caching can only leave more budget, so it never truncates a rollout earlier and never causes a command to time out that would otherwise have completed. The timeout requested by each command (capped at 300 seconds) is unchanged. Only commands whose remaining budget is smaller than their requested timeout are affected. This is the same effect as running the uncached system on faster hardware.

The restriction can be removed with a logical clock. EchoRL already measures the real execution time of each entry for the admission test (Appendix B.5). Charging this recorded time to the task budget on every hit makes the budget consumption of cached and uncached rollouts agree, and it requires no change to the system clock.

We also agree on the scope of the evidence. Figure 3(a–b) covers EgoSchema and SkyRL-SQL only, and the shadow audit was disabled in our experiments. In addition, the shadow audit compares observations and state fingerprints, and it does not cover termination or verifier outcomes. The submission therefore does not contain a paired Terminal-Bench audit under the actual budget rules, and we do not have one to report. In the revision we will (i) add the wall-clock condition to Theorem 1 as an explicit assumption, (ii) describe the logical-clock accounting, and (iii) state this limitation of the Terminal-Bench evidence.

W2 and Q2: Can AGR become stuck on obsolete states?

The failure mode exists in principle. Under the current admission path, a new entry starts with a zero score and cannot displace a positive-score resident (Appendix B.5). A candidate that is rejected for this reason is not admitted later through advantage feedback alone, because it is never resident when its score would be assigned. Space held by positive-score entries is released in three ways.

  1. Score correction on revisits. The score of an entry is recomputed in every iteration in which it is accessed. If the policy has moved away from a state, the rollouts that still reach it receive negative advantages. The entry then turns negative and becomes the first eviction candidate. Only entries that turned positive and are then never visited again remain stale.
  2. Expiration and invalidation. Entries are subject to time-to-live expiration, which applies regardless of the advantage score (Appendix B.5). This bounds how long a stale positive entry can hold space. The submission mentions this mechanism without specifying it, and we will give the full rule and its setting in the revision.
  3. Per-task budgets. Cache keys include the task identity, and capacity is limited per task as well as globally (Appendix C.1.5). Stale entries of one task therefore cannot consume the per-task share of another task.

Figure 3(d) gives indirect evidence on the net effect. AGR and plain LRU share the same hierarchy and budgets and differ only in eviction and admission order. A difference in hit rate between them therefore implies that the cache was under capacity pressure. If blocking by stale positive entries dominated, AGR would fall below plain LRU under that pressure. Instead it is generally higher across epochs and ends at about 36% against 33%. We agree that this is an aggregate outcome.

We will add this failure mode to the limitations and describe two refinements that fit the existing interface. The first is score decay, in which the score of an entry that is not accessed in an iteration is multiplied by a factor below one, so that stale positive entries return to the zero-score group and become evictable. The second is to keep scores on the lightweight metadata of rejected candidates, so that a candidate with positive feedback competes with positive-score residents by recency.

Q3: End-to-end training time and the practical benefit on SQL

The reviewer's assessment of SkyRL-SQL is correct. The table below adds the share of rollout time spent in environment execution, measured without caching. The last column is an estimate from Amdahl's law, 1 / ((1 − f) + f / s), which assumes that inference time is unchanged. It is not a separate measurement.

WorkloadEnv. share of rollout time, f (No Cache)Env. speedup, s (Table 1)Env. time saved per rolloutImplied rollout speedup (estimate)
EgoSchema87%2.1×6.3 s≈1.8×
Terminal-Bench53%5.9×174.1 s≈1.8×
SkyRL-SQL7%8.3×0.10 s≈1.1×

SkyRL-SQL rollouts are dominated by model inference, so the practical training benefit there is small. Its role in the paper is to show that observation-only reuse (L3) preserves training behavior at negligible overhead, not to show a saving in training time. The practical benefit of EchoRL is concentrated in workloads where environment execution takes a large share of the rollout, which are EgoSchema and Terminal-Bench.

These estimates cover the rollout phase. The training loop is synchronous and alternates between collecting a rollout batch and one optimization step (Appendix C.1.1), and EchoRL does not change the optimization step, so the speedup of the full training loop is lower than the rollout speedup. The cost of cache queries, snapshot creation, and restoration is reported separately in Figure 4. We do not have a complete end-to-end wall-time measurement at matched updates that separates inference, optimization, snapshot creation, and waiting. In the revision we will add the time breakdown above, state the role of SkyRL-SQL explicitly, and change the summary claims to refer to environment execution time.

Thank you for the candid and detailed response. I appreciate the clarification of the correctness guarantees and the limitations of AGR. However, the missing correctness audit and unresolved stale-entry blocking leave my main concerns insufficiently addressed, so I am maintaining my rating at 2.

Reviewer_54rwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes agentic RL training more efficient by caching environment states and tool call results. It introduces three layers of caching, of environment, post tool-calls, and post (expensive) deterministic operations. The idea is very simple and natural. The paper shows significant gains on various agentic RL tasks such as Terminal-Bench.

Critical strengths
  • The multi-level caching approach is natural, and the experimental results show significant gains.
  • Advantage-guided retention is neat and makes sense.
  • The paper is generally well-written, and the notation makes it fairly clear what's happening, which is appreciated.
Critical weaknesses
  • The effectiveness of caching depends a lot on the workload and the hardware. This paper could benefit from a more thorough analysis on when this method should be expected to yield gains. Currently, the experiments are fairly basic, showing caching is better than no caching. See questions below for more detailed ideas.
  • Using advantages to prioritize what to retain in the cache is interesting; it would have been nice to see more ablations on different variants of this. For example, is the size of the entry and the cost to compute the entry factor into the priorities? Things that are slower to compute and take less space seem like they should be prioritized even if their advantage is a bit less?
  • The presentation of the method, while precise, could benefit from having a concrete running example just to make it more accessible.
Questions for authors
  • Figure 3 shows that the cached and the no-cache versions obtain different accuracies. Is there any inherent randomness in the evaluation? If not, why do these differ?
  • I have many questions around how the performance improvements vary according to hardware. What if there were 2x or 8x H20s versus 4x H20s? What happens if the amount of memory / disk is changed (either up or down)? What is needed for the IO bandwidth from the cache? Are there multiple types of storage for the different cache levels, and do these correspond to L1, L2, L3 somehow? What is the breakdown in time spent between doing agentic rollouts and tool calls? What if you used a 27B model instead of a 9B model?
  • This method assumes determinism. What happens in settings when tool calls are not deterministic?
  • What if timing information matters for the tool call (e.g., reward associated with how long something takes)? Do you have to manipulate the system time of the environment?
3 comments

We thank the reviewer for the detailed and thoughtful questions.

W1: When should EchoRL be expected to yield gains?

The benefit of a cache hit is ΔC = C_exec − C_reuse (Eq. 7). The gain of EchoRL is therefore governed by four factors.

  1. Prefix overlap. The more rollouts of a task share tool-call prefixes, the more states can be reused.
  2. Execution-to-reuse cost ratio. The gain per hit grows when tools are expensive and restoration is cheap. In Figure 4, 49.4% of full tool calls take longer than the maximum observed latency of every cache operation (query, snapshot build, and restore).
  3. Fraction of deterministic operations. Operations that are not deterministic bypass the cache and yield no gain.
  4. Share of environment time in the rollout. This determines how much of the environment-side speedup is visible end to end.

Our three workloads cover different regimes. The table below combines the environment-side speedup in Table 1 with the share of rollout time spent in environment execution, measured without caching. The last column is an estimate from Amdahl's law, 1 / ((1 − f) + f / s), which assumes that model inference time is unchanged. It is not a separate measurement.

WorkloadEnv. share of rollout time, f (No Cache)Env. speedup, s (Table 1)Implied rollout speedup (estimate)Regime
EgoSchema87%2.1×≈1.8×Environment-dominated; few, expensive tool calls with short prefixes
Terminal-Bench53%5.9×≈1.8×Balanced; many expensive state-changing operations (installs, builds)
SkyRL-SQL7%8.3×≈1.1×Inference-dominated; cheap read-only queries with high overlap

This comparison shows that the two quantities are complementary. SkyRL-SQL has the largest environment-side speedup, but environment execution is only 7% of its rollout time, so the end-to-end effect is small. EgoSchema has the smallest environment-side speedup, but environment execution dominates its rollouts, so the reduction carries over almost entirely. Terminal-Bench benefits on both dimensions. In short, EchoRL is most useful for workloads with expensive, deterministic, and frequently shared environment operations that take a substantial share of rollout time. It gives little benefit when tool calls are cheap (the cost filter with γ = 2 already excludes them), when rollouts rarely share prefixes, or when most operations depend on external mutable services.

Appendix D.1 and D.2 further report results across environment backends (Docker, forkd, AgentENV: 5.5×–5.9×) and model scales (4B, 9B, 27B: 5.6×–6.7×). We will add this analysis and the time breakdown to the revision.

W2: Do entry size and compute cost factor into the retention priority?

They do, but at different stages.

  • Compute cost is used at admission. An entry is cached only if C_exec > γ·C_reuse with γ = 2, and for L2 entries C_reuse includes the snapshot construction time (Sec. 3.3, Appendix B.5). Every resident entry has therefore already passed a cost test.
  • Size acts through the storage budgets, which are enforced globally and per task (Appendix C.1.5).
  • Eviction priority uses only the sign of the advantage score (negative, then zero or unscored, then positive), with LRU inside each group.

We use only the sign because the advantage is a directional signal of future reuse, not a calibrated visitation probability (Sec. 3.3). Multiplying its magnitude by cost and size would require additional normalization and hyperparameters, and the magnitude of GRPO advantages varies across tasks.

We agree that the reviewer's suggestion is a natural refinement. A GreedyDual-Size style priority such as S(e)·C_exec(e)/Size(e) would favor entries that are slow to compute and small to store, and it fits into the same eviction interface. We will discuss this variant in the revision and leave its evaluation to future work.

W3: A concrete running example

We agree. Figure 2 already contains an example (installing and compiling flash-attn on a torch 2.1 base image), and Appendix C.2.3 walks through a Terminal-Bench interaction with its cache eligibility, but the main text does not connect them. In the revision we will use the FlashAttention task as a running example throughout Sec. 3: L1 reuses the task-initialized environment, L2 reuses the environment state after pip install and compilation, L3 reuses the results of read-only inspection commands, and the positive advantage of successful trajectories gives these entries retention priority.

Q1: Why do the cached and no-cache versions obtain different accuracies in Figure 3?

The evaluation itself has no sampling randomness. The difference comes from training. The two curves are two independent RL training runs, and training rollouts are sampled at temperature 0.7. The two runs therefore collect different trajectories and reach slightly different policies. Theorem 1 guarantees that cached and uncached rollouts follow the same distribution, not that individual samples are identical. The gaps in Figure 3 are small, change sign across epochs, and show no systematic trend, which is consistent with run-to-run variation between two independent training runs. We will state this explicitly in the revision.

Q2: How do the improvements vary with hardware?

Number of GPUs (2× or 8× H20). EchoRL removes environment execution, which for SkyRL-SQL and Terminal-Bench runs on the CPU and sandbox side and does not depend on the number of GPUs. The environment-side speedups in Table 1 are therefore expected to be largely unchanged for these two workloads. The end-to-end effect does change. More GPUs make inference faster, so environment execution becomes a larger share of rollout time and the end-to-end benefit of EchoRL increases. Fewer GPUs have the opposite effect.

Memory and disk. The budget determines how many entries can be retained. A smaller budget causes more evictions and a lower hit rate, and this is the regime where the choice of what to retain, and therefore AGR, matters most. A larger budget approaches an unbounded cache, where retention matters less and the hit rate is limited by prefix overlap alone. Our configurations are listed in Appendix C.1.5 (for Terminal-Bench, 100 GiB of RAM and 128 GiB of disk).

IO bandwidth. The IO demand is moderate for two reasons. Snapshots are stored as content-addressed blobs, so identical files are shared across snapshots and written once. Restoration first attempts a copy-on-write reflink and falls back to an ordinary copy only when reflinks are unavailable (Appendix B.4). Figure 4 reports the resulting latency distributions of snapshot build and restore, which are well below the latency of the tool calls they replace.

Storage types and cache levels. The levels are defined by reuse granularity, not by storage medium, but there is a natural correspondence. L3 entries hold only observations, which are small and kept in RAM. L1 and L2 entries hold environment snapshots, which are kept on disk or managed by the native snapshot mechanism of the backend. For example, SkyRL-SQL has only observation entries and is configured with no disk budget.

Breakdown between model rollouts and tool calls. Without caching, environment execution accounts for 87% of rollout time on EgoSchema, 53% on Terminal-Bench, and 7% on SkyRL-SQL. The remainder is model inference. See the table and discussion under W1.

27B instead of 9B. Appendix D.2 reports this setting. On Terminal-Bench with Qwen3.5-27B, EchoRL reduces environment execution time from 317.2 s to 47.3 s per rollout (6.7×), compared with 5.9× for 9B and 5.6× for 4B.

Q3: What happens when tool calls are not deterministic?

Non-deterministic operations are not cached. They execute normally.

  • Operations that involve uncontrolled randomness, external mutable services, the current time, or unrecorded network responses are marked non-cacheable (Sec. 3.2).
  • Once such an operation mutates the environment, EchoRL also disables reuse for all later states of that rollout (Appendix A.1).
  • If the randomness can be controlled, it can be captured in the cache key and the operation becomes reusable.

The cost of non-determinism is therefore a lower hit rate. In the worst case EchoRL degrades to No Cache, and it does not produce an incorrect trajectory. The implementation also provides an optional shadow audit that forks the pre-action state, re-executes the action, and compares both the observation and the state fingerprint. A mismatch invalidates the entry (Appendix B.3).

Q4: What if timing information matters for the tool call?

We do not manipulate the system time of the environment.

  • Time-dependent tool outputs (for example, reading the clock) are uncontrolled inputs, so these operations are non-cacheable and execute normally.
  • Rewards are never cached. Each rollout is graded by the private verifier on its own final environment, and in our three workloads the reward does not depend on execution time.
  • Wall-clock budgets are the one place where timing enters our experiments. Terminal-Bench gives each task a wall-clock agent budget, and a cache hit consumes less of it than real execution. Every step still returns the same observation and state, but a rollout that the budget would cut off without caching may run for more turns with caching, as it would on faster hardware.

When timing matters, the execution time that EchoRL already measures for each entry (Appendix B.5) can be stored and charged as logical elapsed time on a hit. We will describe it and state the wall-clock condition next to Theorem 1 in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.