Probing Interaction Awareness in Trajectory Prediction with Counterfactual Perturbations

Trajectory prediction models for autonomous driving often incorporate neighboring-agent information through social pooling, graph-based interaction modeling, message passing, and attention mechanisms. However, commonly use evaluation metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE) do not reveal whether predictions are actually influenced by surrounding agents. To address this limitation, we introduce a counterfactual sensitivity framework that directly tests interaction dependence by perturbing neighboring-agent trajectories while keeping the target agent unchanged. We apply three perturbation strategies agent removal, spatial displacement, and Gaussian noise to evaluate how predictions respond to controlled changes in neighboring agents. We evaluate LSTM-Interaction, AgentFormer, Trajectron++, and FIERY on the nuScenes dataset using four sensitivity metrics, including the proposed Zero-Reaction Rate (ZRR). The results show that most models exhibit limited response to neighboring-agent perturbations, particularly under realistic geometric and noise-based modifications. Across the evaluated models, Zero-Reaction Rates range from 65% to 99%, indicating that many perturbations produce little or no change in the predicted trajectory. We also observe that sensitivity decreases as scene density increases, suggesting that interaction information may be used less effectively in crowded environments. These findings demonstrate why counterfactual perturbation is necessary as a complementary evaluation tool: prediction accuracy alone does not establish whether a model meaningfully uses neighboring-agent information. Counterfactual sensitivity analysis therefore provides additional insight into how trajectory prediction models utilize interactions.