- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TopoMind combines lane topology prediction and driving-scene question answering in a single model. It uses a shared BEV–language representation and a Semantic-Geometric Query Refinement module (SGQR) to incorporate scene context into lane geometry and connectivity prediction. The authors also assemble approximately 1.19 million topology-oriented question–answer pairs for joint training. Experiments on OpenLane-V2 and OmniDrive-nuScenes evaluate topology prediction and scene understanding, showing improvements over the reported baselines. Ablation studies examine the contributions of language supervision, SGQR, and joint training.
Critical strengths
- The paper combines scene question answering with lane topology prediction, using shared representations and SGQR to incorporate road semantics into lane-query refinement. This design fits the needs of topology reasoning and offers a useful approach to applying language supervision to structured road perception.
- The method outperforms the reported baselines on both lane-segment and centerline tasks. For example, on the Subset-A lane-segment task, mAP improves from 34.4 to 41.6 and the topology score from 31.3 to 33.6, showing benefits for both geometric detection and connectivity prediction.
- Comparisons that remove text, omit SGQR, or directly concatenate queries help identify the sources of improvement. The results support the contributions of language supervision and the fusion strategy, strengthening the evidence for the method’s effectiveness.
Critical weaknesses
-
Section 3.3 describes using ground-truth answer prefixes during training and generated text during inference, but it remains unclear which text states SGQR uses in the topology experiments and whether lane prediction requires answer generation first. This affects the information available to the topology branch, the correspondence between training and inference, and reproducibility. Until this is clarified, I have reservations about how to interpret the main results.
-
The external topology baselines do not use LLMs or VLMs, while the cited Chameleon already uses VLMs for topology reasoning. Different settings may preclude a direct numerical comparison, but the paper needs a more concrete account of what it advances beyond this approach. The scene-understanding evaluation also needs to account for the protocols and results of HERMES and GaussianDWM when making state-of-the-art claims. The internal ablations support the design, but leave uncertainty about its advantage over existing approaches, which limits my assessment of the contribution.
-
The method introduces a 2B language model but does not report end-to-end inference latency, throughput, or memory usage. While the accuracy gains over topology baselines without LLMs or VLMs are clear, the additional computational cost remains unknown. Reporting these measurements under matched hardware and input settings would help assess the accuracy–efficiency tradeoff.
Questions for authors
-
What text is supplied during topology evaluation, and which text hidden states does SGQR use? Is lane prediction performed before answer generation, or does it depend on generated answers? Please clarify the corresponding training and inference settings and the exact procedure used for Tables 1 and 2. This directly affects my assessment of reproducibility and the sources of improvement.
-
What is the main methodological advance over Chameleon? If a direct comparison is inappropriate, which differences in inputs or training settings prevent it? Also, does Table 6 use the same QA evaluation protocol as HERMES and GaussianDWM? If so, would the state-of-the-art claim still hold after including their results?
-
Under the clarified inference procedure, what are the full model’s per-frame latency and peak memory usage? How much overhead does it introduce relative to the variant without the language branch under matched hardware and input settings? This would help assess the computational cost of the reported gains.
I am open to revising my rating if the authors’ response adequately addresses the key concerns above.
Dear Area Chair,
We responded point by point to all three reviews. Our new experiments test whether language helps predict lane graphs and whether learning to predict those graphs helps the model answer topology questions correctly.
1. Do directional questions account for the perception gains? (nT73, qgJk) We added a directional-QA-only run to the Subset-A ablations using our rebuilt LaneSegNet baseline.
Broader scene QA adds 3.1 mAP and 1.7 TOP over directional QA alone. Removing semantic-geometric query refinement (SGQR) lowers mAP/TOP by 2.0/0.7 relative to the full model. Appendix A clarifies how language enters the decoder in Tables 1 and 2. Training uses reference answers with teacher forcing. At inference, the LLM generates answers from questions and BEV tokens, then encodes the dialogue with the BEV tokens. Lane queries attend to the projected, pooled text states and recovered BEV features before predicting geometry and connections.
2. Are topology answers more accurate? (qgJk) On Subset-B validation frames, we scored ego-lane direction by accuracy and connected downstream directions by mean per-frame set F1, which credits partial matches.
Joint training gains 3.65 points in accuracy and 3.81 in F1 on questions with verifiable answers. It also improves Subset-B centerline L-OLS from 46.3 to 54.6 over perception alone (Table 4).
3. What does TopoMind add over Chameleon? (GoVm) We added Chameleon’s published 3-shot result. It infers topology from lanes predicted by a separate detector with an SD map.
TopoMind jointly predicts lanes and connections from cameras without an SD map, exceeding this configuration by 3.75 points in detection and 1.95 in topology.
4. How does understanding compare across model sizes and modalities? (nT73, GoVm) We expanded the OmniDrive-nuScenes comparison beyond OmniDrive-BEV, the closest architectural baseline because both models pass BEV features to the LLM.
C denotes cameras, T text, L LiDAR point clouds, G reconstructed 3D Gaussians and V RGB-D video. With a 1.7B LLM, TopoMind exceeds all three 7B OmniDrive variants and 8B GaussianDWM on CIDEr. Without point-cloud supervision, it comes within 0.006 CIDEr of LiDAR-supervised HERMES.
5. What does the language branch cost? (qgJk, GoVm) Section 4.5 compares full answer generation and lane prediction on Subset-B with perception alone, and reports HERMES on the same A100 hardware.
vLLM speeds up the same TopoMind model 25.3× over HF. TopoMind/vLLM takes 1.379 s/frame versus 17.689 for HERMES/HF and uses 12.13 GiB less peak memory. We will release the training and evaluation code, topology VQA dataset and generation engine.
All three reviews now carry a rating of 3 (Weak acceptance). GoVm raised their rating from 2 to 3 after our response. qgJk confirmed that most concerns had been addressed and maintained their positive rating. nT73 thanked us for the new experiment and requested two further revisions. We revised the abstract to claim state-of-the-art lane perception and competitive scene understanding, explained our Subset-B implementation, and added our perception-only LaneSegNet† baseline to Tables 1 and 2. With the same perception code and preprocessing, full TopoMind gains 4.7 mAP and 5.2 TOP_lsls on Subset-A lane segments and 6.7/8.3 L-OLS on Subsets A/B centerlines.
The authors