acceptodds
Under review as a conference paper at ICLR 2027

Environment-Staleness-Aware Replay for Language Agents

Abstract

Reusing past interactions can make reinforcement learning for large language model agents more efficient, but changes in tool reliability can render those interactions misleading even when the policy remains unchanged. We introduce Environment-Staleness-Aware Replay (ESR) to retain useful historical experience while accounting for changes in the environments that produced it. ESR executes the same actions from restored states in both environment versions and uses a regularized least-squares fit to estimate how the probabilities of tool outcomes have changed. It combines these estimates with policy corrections to reweight complete interaction sequences, excludes replay without sufficient execution evidence, and counts probing and restoration toward the interaction budget. On AppWorld, ESR improves mean post-change task-completion area under the learning curve by 2.44–4.37 points on a 0–100 scale over training with fresh interactions alone across three models and two test splits, with all methods sharing the same total interaction budget. These findings identify environment staleness as a distinct source of replay error and support evidence-based experience reuse in restorable environments with changing tool reliability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, reinforcement learning, experience replay

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LXgDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the environment staleness problem in reinforcement learning for LLM agents: when the reliability of external tools changes, replay gradients from historical interaction trajectories can deviate or even reverse direction, even though the policy itself remains unchanged. The authors propose ESR, which performs paired executions of the same state–action pair in both the source and current environments, estimates changes in tool-outcome probabilities via regularized least-squares density-ratio fitting, and combines the resulting environment ratio with the policy ratio to reweight complete trajectories. An evidence gate excludes tools lacking sufficient observations, and all probing and restoration costs are charged to a shared interaction budget. On AppWorld, ALFWorld, and WebShop, ESR improves AUC by 2.44–4.37 points over fresh-only training under matched budgets, and demonstrates cross-domain transfer without target-domain probing.

Critical strengths
  1. The frozen-policy pre-experiment cleanly shows that tool-reliability drift alone can reverse replay gradients without any policy change.
  2. Improvements hold across three backbones, three benchmarks, and a cross-domain transfer setting.
Critical weaknesses
  1. The method requires full state restoration, invariant tool interfaces, and binary (failure/normal) reliability changes with invariant within-label kernels—conditions that may not hold in real deployment.
Questions for authors
  1. The environment change in your experiments is injected as a single, abrupt reliability drop from 0.7 to 0.5 at a known point (40% of training). In practice, however, tool reliability often drifts gradually, asynchronously across multiple tools, and may not be explicitly observable. Can ESR adapt to such continuous, latent drift? If the version-switch timing is unknown, how should re-fitting be detected and triggered?
  2. Current model assumes that each action triggers at most one affected tool and that reliability depends solely on tool identity. In real agent systems, however, a single action may involve coordinated calls to multiple tools, reliabilities may be interdependent, and several tools may drift simultaneously within one step. Under such multi-tool, state-dependent reliability scenarios, does ESR's one-hot feature encoding and per-tool independent estimation still hold? Would richer interaction features and a joint estimation mechanism be necessary?
4 comments
AuthorsOpenReview ↗

Response to Reviewer LXgD (Part 1/2)

Thank you for the time you spent reviewing our paper and for your valuable feedback on deployment assumptions and realistic drift. We respond below with restoration evidence and experiments on unknown change times, correlated tool outcomes, and state-dependent reliability.

W1. Restoration, fixed interfaces, binary outcomes, and invariant within-label kernels may not hold in deployment.

Response.\color{green}{\mathbf{Response.}} The binary wrapper labels identify reliability branches while retaining the full state and response distribution within each branch. In Eq. (2), y=1y=1 means normal underlying execution, which can include application errors and unsuccessful task outcomes. The kernel GyG_y models these outcomes and their state dependence. Fixed interfaces and invariant within-label kernels make the reliability ratio identifiable.

Evaluated sandboxAudited restoration checks passedPre-sampling anchor exclusionsInvalid probe rate
AppWorld1,500/1,5000.283%0.410%
ALFWorld1,500/1,5000.410%0.539%
WebShop1,500/1,5000.733%0.810%

The audit covers database, interpreter/session, files, clocks, and versioned code, with 500 checked states per seed. All checked states pass, providing evidence for the restoration contract in these sandboxes. The ratio construction also supports finite multi-outcome labels with invariant within-label kernels and source coverage. Q2 tests this extension with four joint outcomes. Theorem 1 establishes gradient recovery under the original reliability model, and Appendices D.8–D.9 explain the roles of representation and support in identifying the correction.


Q1. Can ESR track gradual, asynchronous, latent changes without a known switch time?

Response.\color{green}{\mathbf{Response.}} The window and detector extensions track drift using observed tool outcomes, with true change times hidden. Source cohorts are indexed by round. ESR-Window retains recent paired observations. ESR-ADWIN uses ADWIN's adaptive-window principle [1] to monitor fresh labels for each tool, expire stale observations, and trigger refitting. Monitoring uses fresh labels without reward-based filtering. Window updates preserve common-anchor pairing. True probabilities and change-point identifiers are used only for evaluation.

Frozen-policy diagnostics:

DriftWindow log-ratio MAEADWIN log-ratio MAEADWIN delay: calls / tool labelsADWIN static false alarmsProbe + restore calls per seed: Window / ADWIN
Static0.023±0.0070.029±0.009n/a0 / 90K labels60.0K / 59.4K
Unknown abrupt change0.118±0.0190.061±0.01414.0K / 450n/a60.0K / 59.3K
Gradual change0.063±0.0130.071±0.016n/an/a60.0K / 59.0K
Asynchronous mixed change0.103±0.0180.067±0.014Tool 3: 18.0K / 570n/a60.0K / 59.7K

The 90K static labels pool three seeds. The displayed costs cover diagnostic probing and restoration. The shared fresh monitoring stream and independent evaluation executions are accounted for separately. The results favor a fixed window for smooth drift and adaptive detection for abrupt or asynchronous changes.

End-to-end ALFWorld training under asynchronous mixed drift:

MethodSR-AUCFinal SR (%)Paired AUC gain over Fresh [95% t CI]Total charged callsWall time (h)
Fresh-only53.40±1.2062.75±1.12—996.2K64.2
Policy-only51.07±1.3160.32±1.21−2.33 [−3.27, −1.39]995.9K66.7
ESR-Window55.63±1.2764.93±0.56+2.23 [+1.23, +3.24]994.7K72.1
ESR-ADWIN56.60±1.3965.80±0.75+3.20 [+1.69, +4.71]994.4K73.0

All training probes, restoration, and monitoring costs are charged. Evaluation uses 134 unseen tasks with four trials each. Intervals use three paired seed-level differences. AUC starts at the first change at 30% of the budget. Table 6 uses a separate single-switch protocol starting at 40%. Each round has fixed dynamics and executable source snapshots. Under this setup, ESR-ADWIN improves AUC by 3.20 points, with a paired 95% interval of [1.69, 4.71], demonstrating effective adaptation to the tested unknown drift schedule.

[1] Learning from Time-Changing Data with Adaptive Windowing.

AuthorsOpenReview ↗

Response to Reviewer LXgD (Part 2/2)

Q2. Do correlated multi-tool calls and state-dependent reliability require joint features?

Response.\color{green}{\mathbf{Response.}} Joint outcome features capture the within-action dependence in your example. Across different steps, Eq. (3) accommodates changes in several tools. For two affected calls within one action, the environment ratio is

=\frac{p_\star(y_1,y_2\mid u)}{p_v(y_1,y_2\mid u)}$$ Conditional independence in both environments permits a product of marginal ratios. Sequential conditional factors provide another valid factorization. The composite-action experiment uses source probabilities $(0.25,0.25,0.25,0.25)$ over $(00,01,10,11)$ and target probabilities $(0.4,0.1,0.1,0.4)$. Both tool marginals stay at 0.5, giving unit marginal ratios. The joint ratios are $(1.6,0.4,0.4,1.6)$. | Two-tool setting | Representation | Hold-out joint log-ratio MAE | Gradient cosine | |---|---|---:|---:| | Independent drift | Product of marginal ratios | **0.045±0.011** | **0.969±0.012** | | Independent drift | Four-cell Joint-ESR | 0.064±0.016 | 0.957±0.016 | | Correlation-only drift | Product of marginal ratios | 0.694±0.013 | 0.24±0.13 | | Correlation-only drift | Four-cell Joint-ESR | **0.061±0.018** | **0.935±0.020** | Both fitted estimators use 2,950 valid pairs and 12,000 probe/restoration calls. Each minibatch gradient is paired with an independent fresh-reference gradient to compute cosine. These values are averaged within each seed and summarized across three seeds. The independent control supports marginal factorization when its assumptions hold. Under correlation-only drift, joint encoding reduces MAE from 0.694 to 0.061 and raises cosine from 0.24 to 0.935. Your state-dependent case is tested separately using pre-action context: | Opposing context changes with unchanged pooled reliability | Pooled ESR | Context-ESR | |---|---:|---:| | Hold-out log-ratio MAE | 0.448±0.012 | **0.047±0.014** | | Gradient cosine | 0.45±0.12 | **0.953±0.012** | | Valid pairs / probe + restore calls | 2,940 / 12,000 | 2,940 / 12,000 | These extensions make the representation match the structure of the drift. A joint encoding uses up to $2^m$ outcome cells for $m$ binary calls, with evidence checks applied to each cell. The submitted AppWorld experiment models the top-level execution service. The composite-action experiment directly tests correlated failures across two affected calls, and the context experiment tests state-dependent reliability. Both recover information that pooled marginal features discard.

Thank you for the authors’ response. My concerns have been largely addressed. Given my limited familiarity with this area and the fact that my current score is already positive, I will maintain my score.

We appreciate your recognition of our frozen-policy study isolating environment staleness and the consistent gains across models and benchmarks. Thank you for your thoughtful feedback and positive assessment.

Reviewer_A64zOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Environment-Staleness-Aware Replay (ESR) for reinforcement learning with LLM agents. Even when the agent’s policy stays the same, changes in tool reliability can make past trajectories less useful for training. ESR addresses this by restoring historical states in the source and current environments, replaying the same actions, and using regularized least squares to estimate changes in tool outcome probabilities. It then combines these estimates with policy likelihood ratios to reweight complete trajectories. The authors evaluate ESR on AppWorld, ALFWorld, and WebShop. Under a shared budget that includes fresh interactions, probes, and restoration costs, ESR improves post-change performance over fresh-only training in the controlled experiments. The paper also discusses limits of the method, including missing support and changes to outcomes that are not captured by its reliability model.

Critical strengths
  1. The paper clearly identifies environment staleness as a source of replay error. Its frozen-policy experiments show that changes in tool reliability can reduce gradient alignment and, in some settings, reverse the replay gradient even when the policy does not change.
  2. ESR uses paired executions and a regularized least-squares estimator to correct replay weights. The method does not require an auxiliary neural network. The theoretical results show exact gradient recovery when the stated assumptions hold, and also describe limits caused by missing support and changes within outcome classes.
  3. The experiments account for the cost of collecting probes and restoring states. ESR, fresh-only training, and the other controlled methods share an interaction-call budget, so the reported gains include the calls used for probing and restoration. The paper also reports additional wall-clock cost.
  4. The authors evaluate ESR with three backbones, Qwen2.5-7B, Qwen3-4B, and Llama-3.1-8B, and report results on AppWorld, ALFWorld, and WebShop. The controlled experiments show gains across these settings, though some comparisons have limited statistical power because they use three seeds.
Critical weaknesses
  1. ESR requires executable source versions and full-state restoration for stored anchors. This is a substantial requirement: the state must include all information that affects future transitions, such as application data, interpreter and session state, persistent files, and clocks. Many real web APIs and physical environments may not provide this capability, which limits where ESR can be applied directly.
  2. ESR models reliability changes as depending on tool identity and the binary wrapper outcome. Probe data are pooled by tool and outcome, so the estimator does not capture changes that depend on the particular state or arguments. The paper acknowledges this limitation, but it remains important for environments where reliability changes are state-dependent.
  3. The proposed method cannot recover outcomes that have no support in the source data. Its evidence checks assign zero replay weight when the required source outcomes have not been observed, and the paper states that fresh interaction is needed for changes outside historical support. This limits the benefit of replay under large or structural environment changes.
  4. The paper states that unexpected execution failures outside the wrapper contract invalidate a probe pair, while the incurred costs are still charged. It does not report how often this happens in the experiments. Reporting invalid-pair counts and their share of the probe budget would help readers assess the practical cost of this issue.
Questions for authors
  1. ESR estimates reliability changes using data grouped by tool identity. How might the method be extended to account for changes that depend on the current state or tool arguments?
  2. ESR relies on restoring complete historical states. What parts of the method could still be used in environments where state restoration is unavailable or only partially possible, such as live web services or physical environments?
  3. When the environment changes substantially and historical data provide little useful evidence, could ESR adjust its probe allocation and spend more of the budget on fresh interactions? The paper discusses missing support, but does not evaluate an adaptive rule for changing the probe fraction.
3 comments
AuthorsOpenReview ↗

Response to Reviewer A64z (Part 1/3)

We appreciate the time you devoted to your detailed review and your constructive questions about ESR's assumptions and practical use. Below, we address each point with clarification, restoration statistics, and experiments on conditional estimation and adaptive probing.

W1. Complete restoration limits applicability to live services and physical environments.

Response.\color{green}{\mathbf{Response.}} ESR's paired estimator applies to training environments with executable versions and restorable transition state. Section 4.1 and Appendix A.1 specify these requirements. The audit checks database, interpreter/session, file, clock, and versioned execution state:

SandboxRestoration checks passedPre-sampling anchor exclusionsInvalid probe rate
AppWorld1,500/1,5000.283%0.410%
ALFWorld1,500/1,5000.410%0.539%
WebShop1,500/1,5000.733%0.810%

Each benchmark contributes 500 checked states per seed. All checked states pass, with invalid-probe rates below 1% across the three sandboxes. This provides direct evidence that the restoration requirements are achievable in the evaluated training setting. Our response to Q2 explains how the ratio formulation also applies when only partial restoration or versioned outcome observations are available.


W2. Pooling by tool and binary outcome misses state- or argument-dependent reliability.

Response.\color{green}{\mathbf{Response.}} Conditioning on pre-action context resolves the state dependence you describe. Appendix D.2 identifies the population estimate as E[ρ∣ϕ]\mathbb{E}[\rho\mid\phi]. Proposition 2 in Appendix D.8 explains why the feature representation must preserve the variables governing the change.

The context experiment isolates this effect. Both source contexts have normal-execution probability 0.7. Their target probabilities are 0.5 and 0.9, keeping pooled reliability at 0.7 under balanced contexts.

RepresentationSource-weighted hold-out log-ratio MAEGradient cosine
Original ESR: tool × outcome0.448±0.0120.45±0.12
Context-ESR: tool × pre-action context × outcome0.047±0.0140.953±0.012

Both estimators use the same 2,940 valid pairs and 12,000 probe/restoration calls. Context is observed before execution, with task-level cross-fitting. Each minibatch gradient is paired with an independent fresh-reference gradient to compute cosine. These values are averaged within each seed and summarized across seeds. The pooled population ratio is exactly one, with an analytical MAE of 0.44714. Context-ESR reduces the measured MAE by about 90% and raises cosine from 0.45 to 0.953 under stable within-context kernels and source coverage.


W3. How does ESR handle limited or missing source support?

Response.\color{green}{\mathbf{Response.}} ESR uses source-evidence checks to determine which replay corrections are supported by observations. Additional probes can reveal rare source outcomes. Events with zero source probability require fresh target interaction, as established by Theorem 4 in Appendix D.9 for general bounded functionals.

The diagnostic fixes four affected calls per trajectory and compares rare source failures with impossible source failures:

Source → target normal probabilityValid pairs per fitSource failure / normal countsReplay gate coverageTrue unsupported target mass
0.999 → 0.5500 / 3000.0%0%
0.999 → 0.52001 / 1,19916.7%0%
0.999 → 0.51,0005 / 5,99566.7%0%
1.0 → 0.5500 / 3000.0%93.75%
1.0 → 0.52000 / 1,2000.0%93.75%
1.0 → 0.51,0000 / 6,0000.0%93.75%

Counts pool six fits across three seeds and two folds, with equal scoring volumes and nested probe prefixes. Unsupported mass is computed analytically. It is zero when both source outcomes have positive probability. When source failures are impossible, the target mass outside source support is 1−(0.5)4=93.751-(0.5)^4=93.75\\%.

For rare source failures, increasing the probe count raises gate coverage from 0% to 66.7%. With zero source failure probability, the gate consistently excludes unsupported corrections. Under the within-label invariance assumption, this gives a concrete criterion for allocating effort between replay estimation and fresh target interaction. Q3 evaluates an adaptive budget rule based on that evidence.

AuthorsOpenReview ↗

Response to Reviewer A64z (Part 2/3)

W4. Invalid-pair frequency and its share of probe costs are unreported.

Response.\color{green}{\mathbf{Response.}} The audit reports both denominators explicitly. Pooled over three seeds:

SandboxAttempted / valid pairsRestoration failures / execution exceptionsInvalid pairs (%)Calls spent on invalid pairs / all probe + restore calls
AppWorld86,565 / 86,210304 / 51355 (0.410%)508 / 328,822 (0.154%)
ALFWorld73,500 / 73,104319 / 77396 (0.539%)627 / 278,950 (0.225%)
WebShop57,630 / 57,163363 / 104467 (0.810%)779 / 217,900 (0.358%)

Wrapper failure y=0y=0 is a valid observation. Restoration failures and exceptions outside the wrapper contract invalidate a pair, with all incurred calls charged. AppWorld's mean probe/restoration counts round to 57.5K/52.1K, matching Table 12. Table 10 reports a separate frozen-policy diagnostic. Across the three training sandboxes, invalid pairs consume 0.154%–0.358% of the probe/restoration budget, directly quantifying their practical cost.


Q1. How can ESR represent state- and argument-dependent changes?

Response.\color{green}{\mathbf{Response.}} Use an observed pre-action context c(s,a)c(s,a) in the outcome cell. Under invariant within-cell kernels, the ratio is

=\frac{p_\star(y\mid k,c(s,a))}{p_v(y\mid k,c(s,a))}, \qquad \phi=e_{(k,c,y)}.$$ This gives $2KC$ one-hot cells for $C$ contexts, with source-evidence checks and cross-fitting applied to the same cells. | Opposing context changes | Pooled ESR | Context-ESR | |---|---:|---:| | Hold-out log-ratio MAE | 0.448±0.012 | **0.047±0.014** | | Gradient cosine | 0.45±0.12 | **0.953±0.012** | | Valid pairs / probe + restore calls | 2,940 / 12,000 | 2,940 / 12,000 | The experiment uses $K=1,C=2$ and preserves pooled reliability, so the improvement directly measures the value of context. Cosines are computed for minibatch–independent-reference pairs and averaged within each seed. The four-cell encoding recovers the opposing context changes at the same probe cost. Larger context partitions follow the same construction, with evidence checks applied separately to each cell and invariant within-cell kernels as the identification condition. --- > #### **Q2. What remains usable without full state restoration?** $\color{green}{\mathbf{Response.}}$ **The policy/environment ratio factorization remains applicable whenever the transition ratio is identifiable.** Under the tool-only model in Eq. (2), label probabilities depend on tool identity. This gives a direct identification result: correctly versioned historical labels and representative current labels determine $p_\star(y\mid k)/p_v(y\mid k)$, provided source outcomes are covered and observations are retained without outcome-based filtering. Individual states need not be matched in this model. For partial restoration, the restored variables must determine the relevant transition law, or the omitted variables must be correctly marginalized. State-dependent changes can be handled through conditional transition ratios with adequate coverage. DARC [1], for example, estimates these ratios using classifiers that account for differing state–action visitation. ESR's full-state contract in Appendix A.1 provides a direct way to identify the ratio through paired executions at a common anchor. [1] Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers.
AuthorsOpenReview ↗

Response to Reviewer A64z (Part 3/3)

Q3. Can probe allocation adapt when historical data are less useful?

Response.\color{green}{\mathbf{Response.}} Yes. The adaptive quota reallocates calls to fresh interaction when probe evidence is poor. After 200 attempts per active fit/tool, the rule checks valid-pair yield and old-version coverage. Two consecutive rounds with yield below 50% or coverage below 20% reduce α\alpha from 0.10 to 0.05. Every fifth round retries 0.10, and recovered evidence restores the higher quota. Decisions use observed probe statistics. Eq. (9)'s loss mixture stays fixed.

Source → target normal probabilityMethodTest-N AUCTest-C AUCFresh callsProbe + restore callsWall time (h)
0.7 → 0.5Fixed α=0.10\alpha=0.1026.21±0.6810.36±0.46884.6K109.6K72.4
0.7 → 0.5Adaptive26.21±0.6810.36±0.46884.6K109.6K72.4
0.999 → 0.5Fresh-only18.40±1.046.52±0.65996.1K063.0
0.999 → 0.5Fixed α=0.10\alpha=0.1017.25±1.186.07±0.71884.9K109.3K73.1
0.999 → 0.5Fixed α=0.05\alpha=0.0517.85±1.096.31±0.68940.1K54.3K68.2
0.999 → 0.5Adaptive17.93±1.076.36±0.66894.3K99.9K71.9

All totals remain below one million calls. In the normal setting, the adaptive rule retains the fixed quota, so both rows report the same execution path. In the rare-source setting, it reallocates 9.4K calls to fresh interaction and reduces probe/restoration cost by 8.6%. Mean AUC improves by 0.68/0.29 over fixed 0.10.

Fresh-only achieves the highest mean AUC in this extreme setting. The adaptive and fixed-0.05 results are close, with mean differences of 0.08/0.05. The practical benefit of adaptation is that it retains the standard allocation when evidence is sufficient and directs more calls to fresh data when coverage deteriorates.

Reviewer_9bJROpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper targets a realistic problem in reinforcement learning agent-training pipelines. To reduce the generation cost of sampling trajectories, experience replay has become necessary. However, the environment could be shifting when doing on-policy training. For example, the reliability of the tools could change. To address this, the paper proposes ESR (ENVIRONMENT-STALENESS-AWARE), which reweights the trajectory loss based on differences in tool reliability between old and new environments, so the replay strategy does not falsely reuse patterns from failed tools. The optimization process is relatively light, while the experiments have shown its efficacy. However, certain problems about efficiency and the task settings still leave me with some questions.

Critical strengths
  1. The problem that this work focuses on is quite important. Although the paper focuses on the scenario of an unreliable tool, the staleness of the environment is rather important.
  2. The method is quite straightforward yet effective in this scenario.
  3. The experiments in PRELIMINARY STUDY are well designed and quite supportive of the motivation of the work.
Critical weaknesses
  1. The major concern lies in the task setting. The training of the method requires executable source environment versions and restoration of all transition-relevant state. Staleness of the environment is the core motivation, in which case memorising an executable old environment is simply impossible.
  2. The reweighting strategy seems to include multiple processes. The time and memroy overhead caused by these processes should be evaluated.
  3. The scope is specifically designed for tool-use staleness, while the method can't solve more general problems of environment staleness.
Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9bJR (Part 1/2)

Thank you for your time and thoughtful feedback on the feasibility, efficiency, and scope of ESR. We address your concerns below with restoration audits, resource measurements, and experiments on broader forms of drift.

W1. An outdated environment cannot remain executable.

Response.\color{green}{\mathbf{Response.}} An archived environment remains executable after its transition distribution becomes outdated. Section 4.1 studies versionable, restorable training environments, where the source container is retained as target reliability changes. The AppWorld experiment restores the database, interpreter/session state, persistent files, logical clocks, and versioned execution code.

The restoration audit covers 500 states per seed and benchmark, using state checksums and fixed-randomness single-step checks. Paired probes use independent subsequent randomness.

SandboxAudited states passing restoration checksAnchors excluded before samplingInvalid pairs / attempted pairs
AppWorld1,500/1,5001,275/450,000355/86,565 (0.410%)
ALFWorld1,500/1,5001,600/390,000396/73,500 (0.539%)
WebShop1,500/1,5001,980/270,000467/57,630 (0.810%)

All 1,500 sampled states per benchmark pass the restoration checks, and invalid-pair rates stay below 1%. These results demonstrate that source-version access is practical in the evaluated training sandboxes. Retaining accessible source dynamics is also an established setting in off-dynamics learning, including DARC [1].

[1] Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers.


W2. The multistage correction requires time and memory measurements.

Response.\color{green}{\mathbf{Response.}} Table 12 includes probing and restoration in the call budget and reports wall time. The following profile separates these costs from fitting and measures memory on the same eight-H100 system:

MethodRestore / probe / fit time (h)Total wall time (h)GPU-hoursCharged callsTest-N / Test-C AUC
Fresh-only0 / 0 / 062.3418.6996.4K21.84±0.93 / 7.92±0.61
Policy-only0 / 0 / 064.9432.7995.8K19.16±1.12 / 6.58±0.69
Count-ratio2.90 / 3.80 / 0.0370.8445.1995.1K25.54±0.72 / 9.87±0.49
ESR3.10 / 3.90 / 0.0472.4451.3994.2K26.21±0.68 / 10.36±0.46
MethodPeak GPU memory per device (GB)Aggregate peak CPU RSS (GB)Peak snapshot RAM / disk (GB)
Fresh-only71.61183.0 / 27.6
Policy-only72.01273.2 / 29.8
Count-ratio72.51713.6 / 31.4
ESR72.81813.9 / 33.1

Phase times are mutually exclusive. GPU memory is the maximum per-device peak. All methods retain snapshots through the shared rollout format. ESR's fitting takes 0.04 h, consistent with the O(K)O(K) solve in Appendix A.2. Its moment storage scales as O(VK)O(VK), with snapshot storage reported separately above.

Relative to Fresh-only, ESR gains 4.37/2.44 AUC points with 16.2% more wall time and 7.8% more GPU-hours under the shared environment-call budget. Its 994.2K calls include 57.5K probe and 52.1K restoration calls. In Eq. (7), α=0.10\alpha=0.10 sets the budget reservation. The measured counts include all calls incurred during probing and restoration.

AuthorsOpenReview ↗

Response to Reviewer 9bJR (Part 2/2)

W3. How broadly does ESR address environment staleness?

Response.\color{green}{\mathbf{Response.}} An on-policy trajectory can still produce a stale gradient when the environment changes. Theorem 1 shows that policy-only correction recovers the source-environment gradient. Joint policy and environment correction recovers the current-environment gradient under the stated assumptions. Proposition 1 establishes that these directions can reverse. Table 10 isolates this effect with policy ratios fixed to one: long-trajectory cosine improves from 0.31 to 0.931.

This insight motivates correcting the replay distribution for changes in both policy and dynamics. The binary reliability model provides a tractable estimator. The following extensions test context dependence, correlated outcomes, and unknown change times:

Additional settingComparatorESR extensionResult
Opposing context-dependent changesPooled tool–outcome ESRContext-ESRLog-ratio MAE: 0.448±0.012 → 0.047±0.014
Correlation-only change within a two-tool actionProduct of marginal ratiosJoint-ESRJoint log-ratio MAE: 0.694±0.013 → 0.061±0.018
Unknown asynchronous drift, ALFWorldFresh-onlyESR-ADWINSR-AUC: 53.40±1.20 → 56.60±1.39

The first two experiments isolate representation effects with a frozen policy. The third evaluates training under the same one-million-call cap, using its own drift schedule and AUC interval. Together, they demonstrate how ESR extends to conditional reliability, dependence between tools, and unknown drift timing. These extensions retain identifiable transition ratios and source coverage, whose roles are formalized in Appendices D.8–D.9.

I appreciate the authors' efforts in reporting the new results. I am inclined to keep my original score, and raise confidence, which reflects a positive assessment of this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.