TCMA: Training-Free Causal Online Memory Augmentation for Black-Box Pedestrian Trajectory Prediction

A deployed trajectory predictor accumulates information unavailable during training, namely the realized futures of its earlier predictions in the same environment. Existing approaches for exploiting such delayed feedback typically update the base model or train auxiliary memory modules, which may require internal access, gradients, or additional training and can be difficult in black-box or resource-constrained deployment. We therefore study whether completed trajectories can improve the quality of a frozen predictor's candidate set under a fixed output budget. Our key insight is that only trajectories whose futures have fully elapsed are causally admissible as memory. We introduce Training-free Causal Memory Augmentation (TCMA), an output-level adapter that retrieves such trajectories and replaces candidate slots with low historical win rates. To capture cross-scene motion patterns and within-recording recurrence, TCMA combines a scene-invariant egocentric channel with a scene-aware world-frame channel, both tuned online from elapsed labels. We analyze candidate replacement by decomposing its effect into retrieval benefit and displacement harm, and provide finite-sample risk bounds under explicit sampling and selection assumptions. On ETH-UCY, our prospective evaluation across seven predictors reports mean relative reductions of 6.2% in minADE@20 and 8.5% in minFDE@20. These results support output-level adaptation as a promising direction for reusing deployment experience, turning delayed outcomes into causally safe inference-time memory for deployed black-box predictors.