STEM: Selective Temporal Evidence Memory for Online Vectorized HD Mapping

Online vectorized HD mapping incrementally constructs structured map elements from streaming observations. Beyond per-frame accuracy, a practical system should maintain stable predictions across frames. Existing temporal mappers propagate or aggregate historical representations, but memories within a persistent track can have heterogeneous relations to the current query. They do not explicitly separate relation strength, which determines whether a memory is selected, from relation polarity, which determines how selected evidence enters the update. We propose Selective Temporal Evidence Memory (STEM), centered on an inference-time memory operator that makes this distinction. Triple Correlation Aggregation (TCA) accumulates complementary content, task-specific correspondence, and geometric-temporal context evidence into a signed query-memory score. Its magnitude drives sparse memory selection, while its polarity determines branch-specific integration into a query-conditioned residual update. Trackwise Contrastive Regularization (TCR) is a training-only contrastive regularizer on TCA's shared projected query-memory embeddings: it promotes consistency among score-positive same-track pairs and separates confusable cross-track embeddings. Experiments on nuScenes and Argoverse 2 demonstrate gains in mapping accuracy and temporal stability with comparable inference efficiency.