Offline Policy Learning from Outcome-Only Feedback: Identification and the Price of Logging Memory

We study offline reinforcement learning when each logged episode records the full trajectory but only one noisy scalar outcome, with no per-step rewards. Under history-dependent logging, state–action coverage alone may not determine whether a target policy can be evaluated, because the data reveal rewards only through sums along logged trajectories. With known dynamics, we show that a policy value is identified when its occupancy lies in the span of logged trajectory features; for interior tabular rewards, this condition is also necessary, and when point identification fails the value admits a sharp identified interval. The coefficient $C_{\mathrm{out}}(\pi ;\mu )=d^{\pi \top }{\Sigma }_{\mu }^{†}{d}^{\pi }$ is the exact additive change-of-measure constant and quantifies finite-sample difficulty. History-dependent logging separates this trajectory geometry from ordinary state–action coverage. Two loggers can have identical one-step occupancies while combining those visits into different trajectories, changing both identification and $C_{\mathrm{out}}$. We quantify this effect through an order-necessary distortion factor ${\kappa }_{\mathrm{mem}}$ and a target-specific refinement ${\kappa }_{\mathrm{rep}}$. With known dynamics, we propose a projected, box-constrained pessimistic learner whose expected regret depends on $C_{\mathrm{out}}$ of an optimal policy without requiring a global minimum-eigenvalue threshold; block splitting gives a corresponding high-probability guarantee. On a temporal direct-sum family, $C_{\mathrm{out}}^{\star }-1=H{\kappa }_{\mathrm{mem}}$ exactly, with complementary Gaussian and missing-exposure lower-bound mechanisms. For general even horizons, a parallel-state construction gives the minimax rate $\Theta (\max \{\sigma ,\sqrt{H}\}\sqrt{dC/n})$ for the stated high-dimensional tabular class with $C_{\mathrm{out}}^{\star }\le C$ under the non-clipping conditions. Matched-marginal experiments on compiler-derived additive geometry and at the Markov logging boundary confirm that changing cross-time dependence changes outcome geometry and learning difficulty.