From Replay to Natural Arrival: Identifying Agent Action Effects

Replay enables researchers to evaluate agent action repairs by reconstructing a trajectory and comparing a replacement action with the agent's own continuation. Whether this comparison captures the repair's benefit during natural execution, however, depends on what reconstruction preserves and what the experiment observes. We study when replay evidence identifies natural-arrival action effects. For an observation setting containing natural repair outcomes and replay repair/control outcomes, but no natural policy-control outcomes, we derive a sharp bias interval of width 1-α_N under an action–response factorization assumption, where α_N is the natural probability of selecting the repair. Additional replay samples cannot resolve this missing-control uncertainty. We introduce Replay Transport Audit (RTA), which adds independent live-control measurements to identify the bias under stated within-context sampling conditions, without requiring that factorization. In ALFWorld, complete replay agrees with natural-arrival effects within an absolute panel-level tolerance of 0.10. In WebShop, removing a backend option shifts the repair-versus-alternative effect by −0.165 on a [0,1] score scale despite identical visible inputs. In a separate calibrated-pool training study, live-audited corrections improve success over RL-only by 3.47 and 14.35 percentage points on Gemma4-31B and Qwen3.6-27B, respectively. These findings clarify which action-effect claims replay supports and which require additional live observations or assumptions about reconstruction.