acceptodds
Withdrawn from ICLR 2027

From Fragments to Global Maps: Learning Vectorized Map Aggregation with Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large-scale vectorized HD maps provide structured road information that is essential for perception, localization, and planning in autonomous driving. Constructing such maps requires aggregating noisy, fragmented, and overlapping local predictions collected along a vehicle trajectory into a coherent global map. Existing aggregation methods typically rely on hand-crafted rules for fragment association and refinement. However, a fixed set of thresholds cannot effectively handle variations in road structures and prediction errors, often requiring detector-specific tuning or manual adjustment. To address this limitation, we propose MapMergeLLM, a data-driven framework that formulates vectorized map aggregation as conditional sequence generation with a large language model. Given serialized local vectorized maps, our model directly predicts the aggregated global map polylines. To reduce dependence on any particular upstream detector, we train the model on synthetic local maps generated from clean vector maps using corruptions that simulate representative prediction errors. We further introduce a coordinate tokenizer with geometry-aware pretraining to precisely represent map coordinates. In addition, we propose a line-level association loss that explicitly supervises correspondences between local observations of the same map element. Experiments on Argoverse2 and nuScenes using multiple recent upstream detectors demonstrate that MapMergeLLM substantially outperforms heuristic and optimization-based aggregation baselines without detector-specific retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vectorized HD Map, Global Map Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zem9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the aggregation of noisy, fragmented local map element detections into coherent vectorized maps. The authors observe that existing methods such as adopting fixed association thresholds either leave fragments unmerged or merge neighboring lanes, while fixed smoothing rules may ignore noise or remove geometric details. The proposed MapMergeLLM learns aggregation as sequence generation, using corrupted clean maps, coordinate tokens, geometric pretraining and a line-level association objective.

Critical strengths
  • The motivation of this paper is quite clear in identifying a stubborn difficulty in map construction, and the typical failure patterns are illustrated in Figure 1. The porposed approach of learning context-dependent aggregation is reasonable since lane spacing, curvature and prediction errors vary within a scene. Synthetic corruption training also offers a plausible way to reduce dependence on the output distribution of the upstream detector.
  • The main experiments support the usefulness of the proposed pipeline, with higher F1 scores reported than the baselines VMA and MonoLaM aggregators on both the Argoverse2 and nuScenes benchmarks. In the ablation experiments, the authors also show that coordinate representation and pretraining do improve performance.
Critical weaknesses
  • While the proposed approach does improve upon baselines, the rationale and benefits for using a large language model to perform aggregation remain unclear. There is no learned non-LLM aggregator model trainined with the same serialization and training data, so the evidence does not clearly establish the benefits of a LLM aggregator against alternative light-weight aggregator models.
  • The overall robustness and practicality of the method is also questionable. The benchmarks assume no pose noises which is impractical. Secondly, although an LLM‑aggregator can be highly effective, it often demands a dedicated processing unit, and may therefore be computationally prohibitive.
Questions for authors
  • How were the baseline aggregator parameters selected for VMA and MonoLaM? Were they tuned separately for each detector using validation data?
  • What are the decoding time and memory requirements of the LLM aggregator compared to the baselines?
Reviewer_aqdJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MapMergeLLM for aggregating aligned, noisy local vectorized maps into scene-level global maps. It combines a LoRA-adapted Qwen3.5-4B backbone with synthetic map corruptions, dedicated coordinate tokens, coordinate-token pretraining, and a line-level contrastive loss. Experiments on Argoverse2 and nuScenes using MapTracker and StreamMapNet predictions show consistent F1 improvements over VMA and MonoLaM, particularly for road boundaries.

Critical strengths
  1. The paper addresses a practical mapping problem, with clear motivation and informative qualitative examples of fragmentation, duplication, and geometric distortion.
  2. Synthetic training transfers successfully to real detector outputs. Within each dataset, the same aggregation model handles both evaluated detectors without detector-specific retraining; Table 4 also favors synthetic training in most settings.
  3. Table 1 shows consistent improvements across all evaluated dataset/detector/category combinations. Table 2 supports the usefulness of the proposed tokenization and training pipeline.
Critical weaknesses
  1. All external baselines are non-learning methods, so the experiments do not separate the benefit of the pretrained Qwen3.5-4B backbone from that of simply learning aggregation on the proposed synthetic corruption data. Meanwhile, the paper omits inference latency, peak GPU memory, and per-scene input/output token counts needed to evaluate the accuracy–cost trade-off of a 4B autoregressive model against lightweight learned or optimization-based aggregators. In addition, Table 2 removes geometric-task and map-sequence pretraining together (and does not clarify whether the added coordinate parameters EcE_c remain frozen as in Stage 2 or become trainable in w/o Pretrain), leaving the exact source of the pretraining gain unclear.

  2. As defined in Section 4.1 (Lines 388–390), a prediction matches a ground-truth polyline if >75%>75\% of the ground-truth points lie within distance dd of the prediction, without a symmetric prediction-to-GT coverage constraint. Under this one-way criterion, a predicted polyline that covers a GT segment and then hallucinates a long erroneous extension—a failure mode explicitly observed in the autoregressive ablations in Appendix Fig. 5(b, d)—still counts as a true positive without penalty. Complementary bidirectional geometric metrics or endpoint/extension error measurements are needed to verify true geometric accuracy.

  3. Neither Section 4.1 nor Appendix A.1 specifies the exact train/val/test splits, whether geographically disjoint splits were used, or how frames are sampled at inference time (eg, all frames vs. the 15 m training crop interval). Because the default nuScenes and Argoverse2 splits contain substantial geographic overlap and MapMergeLLM synthesizes training data directly from clean ground-truth maps, overlapping splits would expose test-scene road geometries during training. The exact sampling ranges for the four corruption operators (πT,σjit,A,λ\pi_T, \sigma_{\text{jit}}, A, \lambda) and training budgets for both stages are also omitted.

Questions for authors

see weaknesses.

Reviewer_AdxYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MapMergeLLM, an LLM (they used Qwen3.5-4B) that generates a vectorised HD map. They show that the LLM can take a noisy, fragmented vectorised map as input and generate a more accurate global vectorised map. While this is not the first use of LLMs for doing such autonomous vehicle tasks (MapGPT and DriveDreamer-2), they may be innovating in aggregating local map fragments into a global map. The paper suffers from several experimental shortcomings. No computational overhead is reported. The 5% improvement may stem from heavy LLM integration rather than any fundamental improvement. There is no justification for this.

Critical strengths

This paper proposes MapMergeLLM, an LLM (they used Qwen3.5-4B) that generates a vectorised HD map. They show that the LLM can take a noisy, fragmented vectorised map as input and generate a more accurate global vectorised map. While this is not the first use of LLMs for doing such autonomous vehicle tasks (MapGPT and DriveDreamer-2), they may be innovating in aggregating local map fragments into a global map.

Critical weaknesses

The novelty is incremental: the paper proposes an LLM-based map aggregation framework, but prior work has already used LLMs (though perhaps not specifically to generate accurate global HD maps from noisy, fragmented maps) for lane/map generation, such as autoregressive map generation models like MapGPT and DriveDreamer-2.

Syntactic corruption of the maps may not reflect the real-world scenario. Need better justification for why this syntactic corruption does not induce bias in favour of their method over others.

LLM-based approaches introduce computational overhead; how scalable are these methods? This is especially relevant given the 4 to 5% accuracy increase compared with other methods in the paper. There is no computational overhead-related metric reported.

Why does the main paper compare only two methods? Why only two types of tasks?

Questions for authors

Weekness os writen as questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.