GCD: Correcting Hidden-State Bias in Off-Policy Agentic RL

Off-policy reinforcement learning (RL) for large language models bottlenecks at the rollout stage: every parameter update invalidates the key-value (KV) cache of in-flight agent trajectories, forcing either expensive recomputation or staleness that destabilizes training. Recent advances such as M2PO and asynchronous RLHF tolerate token-level staleness by reweighting the policy gradient with second-moment importance corrections, but they implicitly assume that a stale rollout is sampled from a well-defined behavior policy ${\pi }_{{\theta }_{\text{old}}}$. We show that this assumption is silently violated by every modern partial-rollout system: once a stale KV cache is reused under updated parameters ${\theta }_{\text{new}}$, the resulting token distribution is a hybrid ${\pi }_{\text{hyb}}$ whose attention projections consume stale keys and values while its MLP, output, and gating projections use the new weights. The token-level importance weight $w_t={\pi }_{{\theta }_{\text{new}}}(y_t\mid x_{<t})/{\pi }_{{\theta }_{\text{old}}}(y_t\mid x_{<t})$ corrects the sampling distribution but not this hidden-state-level mismatch, leaving an uncorrected source of policy-gradient bias that grows with both staleness and weight-update magnitude. We introduce Gradient-Coupled KV Drift (GCD), an exception-safe correction comprising (i) a high-probability concentration bound on the drift, with explicit per-layer terms for full-attention and Gated DeltaNet layers; (ii) a tail-event detector that flags inputs outside the bound’s regime of validity; (iii) a composite safety guarantee that treats the bias correction as trustworthy only when both the analytical estimate and the detector agree; and (iv) a variance-preservation theorem showing that GCD composes with the M2PO trust region without re-tuning. On Qwen3.5-9B trained with GRPO on MATH-500 and $\tau$-bench, GCD enables stable training at staleness 512 (a regime where M2PO collapses), reduces rollout wall-clock by 38% versus full recomputation, and lifts $\tau$-bench retail pass${}^4$ by 5.7 points over the M2PO baseline.