acceptodds
Under review as a conference paper at ICLR 2027

STEM: Selective Temporal Evidence Memory for Online Vectorized HD Mapping

Abstract

Online vectorized HD mapping incrementally constructs structured map elements from streaming observations. Beyond per-frame accuracy, a practical system should maintain stable predictions across frames. Existing temporal mappers propagate or aggregate historical representations, but memories within a persistent track can have heterogeneous relations to the current query. They do not explicitly separate relation strength, which determines whether a memory is selected, from relation polarity, which determines how selected evidence enters the update. We propose Selective Temporal Evidence Memory (STEM), centered on an inference-time memory operator that makes this distinction. Triple Correlation Aggregation (TCA) accumulates complementary content, task-specific correspondence, and geometric-temporal context evidence into a signed query-memory score. Its magnitude drives sparse memory selection, while its polarity determines branch-specific integration into a query-conditioned residual update. Trackwise Contrastive Regularization (TCR) is a training-only contrastive regularizer on TCA's shared projected query-memory embeddings: it promotes consistency among score-positive same-track pairs and separates confusable cross-track embeddings. Experiments on nuScenes and Argoverse 2 demonstrate gains in mapping accuracy and temporal stability with comparable inference efficiency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
HD map generation, Auto driving perception

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jfYGOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies temporal stability in online vectorized HD mapping, focusing on how different historical memories within the same tracked instance are used. The proposed STEM framework introduces Triple Correlation Aggregation (TCA) into a MapTracker-style temporal mapping pipeline. TCA sums content similarity, low-dimensional matching similarity, and geometric-temporal context similarity into a signed score. It selects Top-K memories by score magnitude, aggregates them through separate branches according to score polarity, and produces a residual update through query-conditioned gating. During training, Trackwise Contrastive Regularization (TCR) constrains selected same-track positive pairs and separates hard cross-track negatives in a shared projection space. TCR is removed at inference. The paper reports improvements in accuracy and stability on nuScenes and Argoverse 2. It includes correlation-factor and loss ablations, sensitivity analyses for K and the negative-mining threshold, polarity-usage statistics, and a 2-by-2 retrieval-by-aggregation ablation in the appendix. The work has clear practical value, and the distinction between memory retrieval and integration is worth investigating. The main concerns are the internal consistency of the central stability results and whether the proposed mechanism's benefits have been sufficiently separated from auxiliary training and branch capacity.

Critical strengths

S1. The problem is specific, and the evaluation metrics match the objective. The paper evaluates Presence, Localization, Shape, and mAS alongside single-frame mAP. Query-conditioned selection among historical entries in persistent memory is a natural and useful direction for tracking-based mapping. Its value does not depend on claiming comprehensive leaderboard dominance. S2. The central design is supported by direct controlled comparisons. Appendix A.2 and Table 11 report four separately trained combinations of magnitude-based retrieval and polarity-aware aggregation, providing stronger evidence than an ablation that only adds the entire module. With polarity-aware aggregation fixed, magnitude-based retrieval improves mAP/mAS by 0.43/0.80 points. With magnitude-based retrieval fixed, polarity-aware aggregation improves them by 0.52/0.98 points. These results provide substantive support for the main design. S3. The main nuScenes results are encouraging. Relative to the baseline in Table 3, the full method improves mAP/mAS from 72.93/84.57 to 75.00/88.41, corresponding to gains of 2.07/3.84 points. TCA alone also improves mAP/mAS by 0.59/1.59 points, indicating that the inference-time operator's benefit does not depend entirely on TCR. The single-factor and leave-one-out ablations in Table 4 provide additional evidence that the three correlation sources may be complementary. S4. The scope of the methodological and theoretical claims is generally appropriately qualified. The authors explicitly state that score polarity is a learned relation rather than a ground-truth semantic label. The appendix proposition is restricted to an operator-level property with fixed value features and does not claim that the complete model is universally more expressive than attention. These qualifications are appropriate. S5. The deployment cost is attractive. TCR is used only during training. Table 6 reports similar latency for the two methods on the same RTX 4090, supporting comparable inference efficiency. This is practically relevant for continuously operating perception modules. The multi-frame overlays in Figure 4 also show cleaner and more continuous local geometry, although they primarily provide qualitative support.

Critical weaknesses

W1. The central Argoverse 2 mAS result has a verifiable inconsistency with the stated formula and requires priority clarification. Page 7 defines the stability of each matched instance as: $ s_e=p_e\bigl(0.7l_e+0.3h_e\bigr),\qquad 0\le p_e,l_e,h_e\le1.$ If Loc, Shape, and mAS in Table 2 use the same instance sets and the same within-class and across-class averaging, then necessarily mAS≤0.7,Loc+0.3,Shape.\mathrm{mAS}\le0.7,\mathrm{Loc}+0.3,\mathrm{Shape}. The STEM row in Table 2 reports Loc=88.48 and Shape=98.01, yielding an upper bound of 91.339. However, the reported mAS is 93.25, exceeding this bound by 1.911 percentage points. This discrepancy is substantially larger than two-decimal rounding can explain. This argument does not multiply marginal means; it uses a necessary condition obtained by applying the same order-preserving averaging operator to an instance-wise inequality. If the component metrics and mAS use different valid samples, class weights, or matching filters, the authors should specify those conventions and explain their consistency with the stated definition. A table error or an implementation/configuration error is also possible. Until clarified, the claimed 2.98-point mAS improvement on Argoverse 2 cannot serve as reliable evidence for acceptance. This issue does not establish research misconduct or directly demonstrate that TCA is ineffective; it undermines the reliability of a central reported result.

W2. The provenance of the baselines and the experimental settings are insufficiently documented to establish fully fair comparisons. On nuScenes, MapTracker achieves 73.26 mAP in Table 1 but 72.93 in Tables 3 and 6, with noticeably different category-level AP values. The 2.07-point gain stated in the contribution summary corresponds to the latter baseline; the gain relative to Table 1 is 1.74 points. These results could reasonably come from published results and the authors' reproduction, respectively, but this distinction should be explicitly labeled. Both baselines also have exactly the same reported mAS of 84.57, warranting clarification of their evaluation provenance. Table 1 states only that methods are compared under their reported training settings. It does not individually specify the backbone, image resolution, perception range, original versus geographically disjoint split, annotation processing, or training budget. Geographic leakage in online mapping has been explicitly documented, making the split a substantive evaluation choice rather than a minor implementation detail. Lilja PredMapNet, published at WACV 2026, is another closely related method that the paper cites but does not compare quantitatively. Its published numbers alone do not establish that STEM is worse under a fair comparison. Nevertheless, the authors should explain its omission and position their conclusions under comparable configurations.

W3. Table 11 supports the design's effectiveness but does not fully isolate the contribution of the polarity mechanism. The two branches have independent projections W+W^+ and W−W^- and query-conditioned gating. However, the exact formula, parameter count, and training-time TCR positive-sampling rule for the Unified variant are not provided. The comparison may therefore combine differences in branch partitioning, projection capacity, gating, and regularization sampling. Stating that all other configurations are fixed is insufficient to establish that these factors are controlled. Furthermore, the independent projection W−W^- has no sign constraint. Defining W~−=−W−\widetilde W^-=-W^- absorbs the negative branch's explicit subtraction into its projection. Thus, score polarity does determine branch routing, but the output minus sign alone does not guarantee semantic cancellation of erroneous history. The appendix's appropriately qualified fixed-value convex-hull result does not establish that explicit subtraction is indispensable in the complete network. A capacity-matched two-branch control, or a narrower mechanistic interpretation, would address this concern. TCR also contributes substantially: in Table 3, adding it to TCA improves mAP/mAS by 1.48/2.25 points, exceeding the gains from TCA alone. Since temporal instance contrastive learning has precedents, a conventional trackwise contrastive regularizer that approximately matches the projection space and sampling budget while remaining independent of TCA scores would help establish TCR's specific value.

Questions for authors

Refer to Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer jfYG: W1, W2

Thank you for the detailed review. We clarify the Argoverse 2 stability calculation, reconcile the MapTracker baseline across the nuScenes tables, and explain what the aggregation ablations and the two new controls test.

W1. Argoverse 2 mAS: Localization Weight and Evaluation Protocol

We apologize for the incorrect weight stated in the manuscript. It‘s a typo error. We used a localization weight of 0.5 in the Argoverse 2 evaluation and have corrected the manuscript accordingly.

Matching and averaging. We match predictions to ground truth within each frame and pair observations across frames by ground-truth instance ID. For each run, Presence, Loc, Shape, and SI use the same retained pairs, excluding pairs missed in both frames. Each class mean uses the retained pair count shown below. SI applies Presence to the equally weighted Loc and Shape for each pair before averaging. Each reported mean weights the classes equally.

The table gives the class SI and valid pair counts for STEM and MapTracker in Table 2.

CategorySTEM SISTEM pairsMapTracker SIMapTracker pairs
Ped. crossing94.2180390.31880
Divider93.221,92889.972,108
Boundary92.311,50590.531,601
Class mean / total pairs93.254,23690.274,589

On Argoverse 2, STEM achieves 93.25 mAS, an improvement of 2.98 points over MapTracker (90.27 mAS).

W2. MapTracker Baseline and Comparison Protocol

On nuScenes, the 72.93 mAP entry in the original Tables 3 and 6 was the MapTracker result reported in the AMap paper, whereas 73.26 mAP in Table 1 came from our MapTracker reproduction. The 84.57 mAS was computed using our reproduced model. We have unified the reporting around this reproduction and updated the corresponding mAP gain to +1.74.

The compared methods use ResNet-50, the same input resolution and perception range, and consistent dataset splits and annotation processing, while retaining their method-specific training schedules. Our comparison evaluates temporal mapping under the common original scene-based split.

PredMapNet is discussed in Related Work, but its implementation is not publicly available for a same-protocol reproduction. Its rasterized map history and future guidance also differ from STEM’s query-conditioned selection and integration of individual same-track vector memories.

Thank you for your answer, this will help me improve my rating. As I mentioned earlier, it has been clearly documented that the online maps in nuScenes and Argoverse 2 have geolocation leakage issues. Therefore, even if this paper is ultimately accepted, I strongly recommend that you include experimental results on nuS and AV2 from the “new split”; this will contribute to the community's progress. You can refer to: https://arxiv.org/pdf/2610.01905, https://ieeexplore.ieee.org/abstract/document/10658066.

AuthorsOpenReview ↗

Thanks for your timely information.

Thanks for your following up. We will share the result here and update this result in paper once they are available. Based on the current status of the GPU, we should be able to complete the two sets of experiments for MapTracker and STEM on the new split by the 16th.

Thank you to the authors for the further clarification. I would like to add one more point: I believe the authors' current experiments and analysis are already quite sufficient, and the experiments on the new split are merely suggested as an extension. I look forward to the authors completing these experiments to further strengthen the persuasiveness of the work. I also look forward to the authors' future work.

Reviewer_maEjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STEM, a temporal memory framework for online vectorized HD mapping based on MapTracker. It introduces Triple Correlation Aggregation (TCA) to select historical memories by correlation magnitude and integrate them according to correlation polarity. Trackwise Contrastive Regularization (TCR) further improves intra-track consistency and cross-track discrimination during training. Experiments on nuScenes and Argoverse 2 show improvements in mapping accuracy and temporal stability with comparable inference efficiency.

Critical strengths
  1. STEM improves both mapping accuracy and temporal stability on nuScenes and Argoverse 2, with comparable inference efficiency to MapTracker.

  2. The ablation studies examine TCA, TCR, and the separate effects of magnitude-based retrieval and polarity-aware integration, providing evidence for the proposed design.

Critical weaknesses
  1. The MapTracker baseline results differ across tables. Table 1 reports 73.26 mAP, while Tables 3 and 6 report 72.93. The reason for this difference is unclear.

  2. On nuScenes, lane divider AP decreases from 73.77 to 70.71 compared with MapTracker. In Table 3, adding TCA alone also reduces lane divider AP from 75.77 to 69.45, which is not sufficiently explained.

Questions for authors
  1. Why does MapTracker achieve 73.26 mAP in Table 1 but 72.93 in Tables 3 and 6? Were these results obtained under different experimental settings?

  2. Why does STEM reduce lane divider AP on nuScenes despite improving overall mAP? In particular, what causes the larger drop in the TCA-only variant?

2 comments
AuthorsOpenReview ↗

Response to maEj:, Q1, Q2

Thank you for pointing out the differences in the reported MapTracker results and the drop in lane divider AP. We clarify the source of each baseline value and report where full STEM misses dividers despite improving overall mapping accuracy and stability.

Q1. MapTracker Baseline Across Tables

The two nuScenes values came from different sources: 72.93 mAP was the MapTracker result reported in the AMap paper, whereas 73.26 mAP was our reproduction. The 84.57 mAS was evaluated using our reproduced model. To avoid confusion, we have now uniformly updated the main text to report our results from MapTracker (73.26 mAP and 84.57 mAS).。

Q2. Lane Divider Accuracy and Failure Cases

Why does divider AP decrease, and why is the reported drop larger for TCA-only?

Thank you for highlighting the category-specific results. On nuScenes, STEM improves pedestrian-crossing AP by 6.62 points and road-boundary AP by 1.63 points. These gains outweigh the 3.06-point drop in divider AP, yielding a 1.74-point improvement in overall mAP.

Where the divider AP gap appears. Our precision–recall analysis locates the gap primarily in the high-recall regime. At the strict 0.5 m matching threshold, STEM and MapTracker achieve nearly identical divider AP. At 25% recall, STEM achieves higher precision (91.1% vs. 88.5%), recovering the same fraction of dividers with fewer false positives. At the 1.5 m threshold and 85% recall, STEM includes more unmatched predictions, and its precision drops to 40.8% versus 60.5% for MapTracker. STEM also reaches lower maximum recall at each threshold. Together, these results explain the lower divider AP despite the precision gain at the strict operating point.

Geometric matching thresholdSTEM APMapTracker APSTEM max. recallMapTracker max. recall
0.5 m62.0262.0374.07%75.87%
1.0 m73.0075.0284.56%86.79%
1.5 m79.0281.9188.89%91.38%

What the TCR loss ablation shows. The divider trade-off is concentrated in the cross-track hard-negative comparison. Adding this term reduces divider AP from 72.31 to 70.71, while improving overall mAP from 74.14 to 75.00 and mAS from 82.73 to 88.41. Adding the same-track positive term changes divider AP by only 0.03 points. The hard-negative term explicitly separates embeddings of similar instances on different tracks. For ambiguous fragments of parallel dividers, this separation can change the shared query features used for localization and confidence ranking, providing a plausible mechanism for the observed trade-off.

Why the reported TCA-only variant shows a larger drop. TCR strengthens the temporal representation by encouraging consistency within each track and improving discrimination between similar tracks. In the reported comparison, full STEM reaches 70.71 divider AP versus 69.45 for TCA-only, recovering 1.26 points of divider AP.

Thanks for the response. The baseline difference and the divider results are now clear. Overall, this is nice work, and I look forward to your future research.

Reviewer_4xxLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies online vectorized HD mapping, where predictions should remain stable across frames as well as accurate within each frame. The authors argue that stored memories differ in both their relevance to a current query and the direction of their contribution. Existing temporal fusion does not explicitly separate these two roles. STEM builds on MapTracker and assigns a signed score to each query and its same-track memories. Score magnitude selects a small number of entries. The sign routes them to separate branches, which are combined with opposite signs before updating the query. A training-only contrastive loss aligns positively scored same-track embeddings and separates confusable cross-track embeddings. On nuScenes and Argoverse 2, the paper reports improved mAP and temporal stability, with comparable measured inference speed on nuScenes.

Critical strengths

A new and simple way to read temporal memory. Letting one score's magnitude select memory entries and its sign route them is cleanly stated, and as a per-track memory read-out I found no direct counterpart in recent mapping work. It adds no measurable inference cost (Table 6), and the separately trained 2×2 ablation (Table 11) is the right experiment for separating the two roles, so other temporal mappers could adopt it easily.

It targets stability, not only accuracy. Frame-to-frame stability matters for deployed online mapping, the stability metric of StableHDMap (mAS) is still rarely reported, and the paper reports it on both nuScenes and Argoverse 2. This makes the paper relevant beyond another accuracy gain on MapTracker.

Critical weaknesses

The size of the gain over MapTracker cannot be judged. The paper prints two different nuScenes MapTracker rows with the same mAS and states the source of neither, and its MapTracker stability is about seven mAS points below what StableHDMap reports for MapTracker at a similar mAP (arXiv version). No variance is reported, and the threshold behind the headline mAS was chosen on the same validation split; at other thresholds the mAS gain shrinks or turns negative (Table 7). The mAP gain is positive in every reported setting, but its size is not anchored, and this is what keeps my rating at weak acceptance.

Most of the gain comes from the training loss, and the evidence for the read-out itself is thin. In Table 3 the training-only contrastive loss supplies most of the improvement, and in Table 11 separating magnitude and sign adds less than one mAP point, without variance, over ranking by the raw score with a single branch. The premise that some entries carry "negative" evidence is not directly tested: Table 9 shows that about 35% of selected pairs score negative, not what those entries are. The largest score term ignores memory content and alone comes within 0.7 mAP of the full model, so the sign may mostly reflect an entry's age or relative pose.

Questions for authors

Where does each nuScenes MapTracker row come from, and does STEM start from a trained MapTracker checkpoint? Which stability-protocol settings were used, and does your pipeline reproduce StableHDMap's published MapTracker values from the same checkpoint?

Can you report a same-schedule, multi-seed comparison of MapTracker and STEM on that pipeline?

How is the sign distributed over memory age, ego displacement and association errors, and which of the three score terms determines it?

1 comment
AuthorsOpenReview ↗

Response to 4xxL :, Q1, Q2, Q3

Thank you for highlighting the distinction between memory selection and integration and the importance of temporal stability. We clarify the MapTracker baseline and evaluation protocol, report the variation across three nuScenes comparisons, and examine how polarity relates to memory age, ego displacement, and association correctness.

Q1. MapTracker Baseline, Initialization, and Stability Evaluation

MapTracker baseline. The original Tables 3 and 6 used the 72.93 mAP result reported in the AMap paper. To avoid ambiguity, we now report our reproduced MapTracker results (73.26 mAP and 84.57 mAS) consistently across Tables 1, 3, and 6.

STEM initialization. We ran MapTracker’s BEV pretraining and decoder warm-up stages ourselves, then initialized STEM’s joint training from the resulting checkpoint.

nuScenes stability evaluation. We evaluate dividers, pedestrian crossings, and road boundaries using a 2-frame interval, 50 points per polyline, and a 30 × 60 m evaluation region. We use a localization weight of 0.5 to compute mAS.

Comparison with StableHDMap. We were unable to reproduce StableHDMap’s published MapTracker results using the same checkpoints. This is because the publicly available version does not specify how the MapTracker predictions used to generate these results were converted into the evaluator’s input format. We requested the processing files from the authors several months ago but did not receive a response. Therefore, we performed the conversion to the evaluator’s input format ourselves, and we will include the code in the final paper to address this gap in StableHDMap’s documentation.

Q2. Results Across Three nuScenes Comparisons

The three reported nuScenes comparisons are listed below. The seed is (0,1,2)

RunMapTracker mAPSTEM mAPMapTracker mASSTEM mAS
073.2675.0084.5788.41
173.1775.3484.8388.09
272.5174.8284.3688.76
Mean ± Std72.98 ± 0.4175.05 ± 0.2684.59 ± 0.2488.42 ± 0.34

The reported differences are 1.74, 2.17, and 2.31 points for mAP and 3.84, 3.26, and 4.40 for mAS. The mean differences are +2.07 ± 0.30 mAP and +3.83 ± 0.57 mAS (mean ± sample SD). STEM is higher on both metrics in all three reported comparisons.

Q3. Polarity Distribution and Contributions of the Three Score Terms

(1) Sign distribution. We group the Top-K selected query–memory pairs on nuScenes by memory age, ego displacement, and association correctness. We determine association correctness by matching to ground truth.

DiagnosticReference groupNegativeComparison groupNegativeDifference
Memory age1 frame2.44%5–8 frames64.93%+62.49 pp
Ego displacement0–1 m9.77%12–24 m65.83%+56.06 pp
Association correctnessCorrect24.45%Incorrect63.79%+39.34 pp

Negative scores are more frequent in the reported older-memory and larger-displacement groups. The rates also differ by 39.34 percentage points between incorrect and correct associations. The age and displacement comparisons are observational.

(2) Which terms affect the sign? The score sums static, matching, and dynamic correlations. In a term-removal diagnostic on the 531,270 selected pairs from Appendix A.1, we keep the selection fixed and subtract one term from each score. Using all selected pairs as the denominator for each term, we obtain:

Term removedNegative → nonnegativeNonnegative → negativeOverall flip rate
Static2,1212,8740.94%
Matching1,06040,1337.75%
Dynamic24,8727,2876.05%

Matching and dynamic removals have different directional effects in this diagnostic. The rates describe changes to the sum on fixed pairs; they do not measure changes in retrieval or performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

The result on the nuscenes new split

We sincerely thank the reviewer for the encouraging comments and constructive suggestions. We have conducted preliminary experiments on the geographically non-overlapping split, with the results shown below:

MethodmAPPed. CrossingDividerBoundary
UIGenMap38.5740.2530.0145.44
MapTracker39.1645.8328.5843.07
STEM (Ours)40.7746.4730.8245.00

STEM achieves the best overall performance, outperforming MapTracker and UIGenMap by 1.61 and 2.20 mAP points, respectively. These preliminary results demonstrate promising geographic generalization. We will include more extensive evaluations on this split in our future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.