- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the environment staleness problem in reinforcement learning for LLM agents: when the reliability of external tools changes, replay gradients from historical interaction trajectories can deviate or even reverse direction, even though the policy itself remains unchanged. The authors propose ESR, which performs paired executions of the same state–action pair in both the source and current environments, estimates changes in tool-outcome probabilities via regularized least-squares density-ratio fitting, and combines the resulting environment ratio with the policy ratio to reweight complete trajectories. An evidence gate excludes tools lacking sufficient observations, and all probing and restoration costs are charged to a shared interaction budget. On AppWorld, ALFWorld, and WebShop, ESR improves AUC by 2.44–4.37 points over fresh-only training under matched budgets, and demonstrates cross-domain transfer without target-domain probing.
Critical strengths
- The frozen-policy pre-experiment cleanly shows that tool-reliability drift alone can reverse replay gradients without any policy change.
- Improvements hold across three backbones, three benchmarks, and a cross-domain transfer setting.
Critical weaknesses
- The method requires full state restoration, invariant tool interfaces, and binary (failure/normal) reliability changes with invariant within-label kernels—conditions that may not hold in real deployment.
Questions for authors
- The environment change in your experiments is injected as a single, abrupt reliability drop from 0.7 to 0.5 at a known point (40% of training). In practice, however, tool reliability often drifts gradually, asynchronously across multiple tools, and may not be explicitly observable. Can ESR adapt to such continuous, latent drift? If the version-switch timing is unknown, how should re-fitting be detected and triggered?
- Current model assumes that each action triggers at most one affected tool and that reliability depends solely on tool identity. In real agent systems, however, a single action may involve coordinated calls to multiple tools, reliabilities may be interdependent, and several tools may drift simultaneously within one step. Under such multi-tool, state-dependent reliability scenarios, does ESR's one-hot feature encoding and per-tool independent estimation still hold? Would richer interaction features and a joint estimation mechanism be necessary?
4 comments
Response to Reviewer LXgD (Part 2/2)
Q2. Do correlated multi-tool calls and state-dependent reliability require joint features?
Joint outcome features capture the within-action dependence in your example. Across different steps, Eq. (3) accommodates changes in several tools. For two affected calls within one action, the environment ratio is
=\frac{p_\star(y_1,y_2\mid u)}{p_v(y_1,y_2\mid u)}$$ Conditional independence in both environments permits a product of marginal ratios. Sequential conditional factors provide another valid factorization. The composite-action experiment uses source probabilities $(0.25,0.25,0.25,0.25)$ over $(00,01,10,11)$ and target probabilities $(0.4,0.1,0.1,0.4)$. Both tool marginals stay at 0.5, giving unit marginal ratios. The joint ratios are $(1.6,0.4,0.4,1.6)$. | Two-tool setting | Representation | Hold-out joint log-ratio MAE | Gradient cosine | |---|---|---:|---:| | Independent drift | Product of marginal ratios | **0.045±0.011** | **0.969±0.012** | | Independent drift | Four-cell Joint-ESR | 0.064±0.016 | 0.957±0.016 | | Correlation-only drift | Product of marginal ratios | 0.694±0.013 | 0.24±0.13 | | Correlation-only drift | Four-cell Joint-ESR | **0.061±0.018** | **0.935±0.020** | Both fitted estimators use 2,950 valid pairs and 12,000 probe/restoration calls. Each minibatch gradient is paired with an independent fresh-reference gradient to compute cosine. These values are averaged within each seed and summarized across three seeds. The independent control supports marginal factorization when its assumptions hold. Under correlation-only drift, joint encoding reduces MAE from 0.694 to 0.061 and raises cosine from 0.24 to 0.935. Your state-dependent case is tested separately using pre-action context: | Opposing context changes with unchanged pooled reliability | Pooled ESR | Context-ESR | |---|---:|---:| | Hold-out log-ratio MAE | 0.448±0.012 | **0.047±0.014** | | Gradient cosine | 0.45±0.12 | **0.953±0.012** | | Valid pairs / probe + restore calls | 2,940 / 12,000 | 2,940 / 12,000 | These extensions make the representation match the structure of the drift. A joint encoding uses up to $2^m$ outcome cells for $m$ binary calls, with evidence checks applied to each cell. The submitted AppWorld experiment models the top-level execution service. The composite-action experiment directly tests correlated failures across two affected calls, and the context experiment tests state-dependent reliability. Both recover information that pooled marginal features discard.Thank you for the authors’ response. My concerns have been largely addressed. Given my limited familiarity with this area and the fact that my current score is already positive, I will maintain my score.
Thank you
We appreciate your recognition of our frozen-policy study isolating environment staleness and the consistent gains across models and benchmarks. Thank you for your thoughtful feedback and positive assessment.
Response to Reviewer LXgD (Part 1/2)
Thank you for the time you spent reviewing our paper and for your valuable feedback on deployment assumptions and realistic drift. We respond below with restoration evidence and experiments on unknown change times, correlated tool outcomes, and state-dependent reliability.
Response. The binary wrapper labels identify reliability branches while retaining the full state and response distribution within each branch. In Eq. (2), y=1 means normal underlying execution, which can include application errors and unsuccessful task outcomes. The kernel Gy models these outcomes and their state dependence. Fixed interfaces and invariant within-label kernels make the reliability ratio identifiable.
The audit covers database, interpreter/session, files, clocks, and versioned code, with 500 checked states per seed. All checked states pass, providing evidence for the restoration contract in these sandboxes. The ratio construction also supports finite multi-outcome labels with invariant within-label kernels and source coverage. Q2 tests this extension with four joint outcomes. Theorem 1 establishes gradient recovery under the original reliability model, and Appendices D.8–D.9 explain the roles of representation and support in identifying the correction.
Response. The window and detector extensions track drift using observed tool outcomes, with true change times hidden. Source cohorts are indexed by round. ESR-Window retains recent paired observations. ESR-ADWIN uses ADWIN's adaptive-window principle [1] to monitor fresh labels for each tool, expire stale observations, and trigger refitting. Monitoring uses fresh labels without reward-based filtering. Window updates preserve common-anchor pairing. True probabilities and change-point identifiers are used only for evaluation.
Frozen-policy diagnostics:
The 90K static labels pool three seeds. The displayed costs cover diagnostic probing and restoration. The shared fresh monitoring stream and independent evaluation executions are accounted for separately. The results favor a fixed window for smooth drift and adaptive detection for abrupt or asynchronous changes.
End-to-end ALFWorld training under asynchronous mixed drift:
All training probes, restoration, and monitoring costs are charged. Evaluation uses 134 unseen tasks with four trials each. Intervals use three paired seed-level differences. AUC starts at the first change at 30% of the budget. Table 6 uses a separate single-switch protocol starting at 40%. Each round has fixed dynamics and executable source snapshots. Under this setup, ESR-ADWIN improves AUC by 3.20 points, with a paired 95% interval of [1.69, 4.71], demonstrating effective adaptation to the tested unknown drift schedule.
[1] Learning from Time-Changing Data with Adaptive Windowing.