Latent-Type Posterior Conditioning for Heterogeneous Multi-Agent Trajectory Prediction under Noisy and Missing Type Labels

Heterogeneous trajectory predictors condition each agent’s representation on a semantic type label (pedestrian, cyclist, vehicle) and treat it as ground truth. In deployment the label is a perception output that may be wrong, missing, or in-consistent with the agent’s motion, which raises the question of when a predictor should rely on it. We propose posterior type conditioning: a dynamics-based type classifier and a learned, context-dependent noisy-label channel are combined by Bayes’ rule into a per-agent posterior over types, which conditions the predictor. Both are trained jointly from the prediction loss and the label likelihood, without supervision of label reliability. On a controlled benchmark and three real-world datasets spanning metric- and pixel-space trajectories and hand-annotated and perception-derived labels (COSMOS, Stanford Drone, Argoverse 2), we find that the value of the label is strongly class-dependent: relative to a type-agnostic model it improves prediction for a behaviorally distinct minority class by 8–15% (pedestrians among vehicles) and by 1–3% where motion already identifies the type. Naive conditioning degrades by 11–34% at 50% label corruption; label-noise augmentation removes most of this loss but forfeits part of the minority-class gain. Posterior conditioning retains most of that gain, degrades more gracefully, and is statistically indistinguishable from the best model on held-out scenes, cities, and recording dates. Its per-agent trust score detects corrupted labels with AUROC 0.91–0.97 on two of three datasets and recovers interpretable label–motion relationships, such as crediting a cyclist label only at cycling speeds.