acceptodds
Under review as a conference paper at ICLR 2027

TopoMind: Unifying Lane Topology Reasoning and Understanding on Driving Scenes

Abstract

Lane topology reasoning provides geometric and relational constraints essential for autonomous driving. Existing methods, however, treat it primarily as a perception problem, detecting lane elements and then inferring connectivity without understanding topology-relevant scene context. We present TopoMind, to our knowledge the first unified framework for lane topology perception and topology-oriented scene understanding. TopoMind integrates lane-topology perception and topology-oriented scene understanding within a shared BEV–language architecture, where a large language model jointly processes metric BEV and text tokens and Semantic-Geometric Query Refinement couples the resulting multimodal context with geometry-anchored lane queries for topology reasoning. To supervise this capability, we assemble a topology-oriented VQA corpus comprising approximately 1.19 million question–answer pairs, covering road layout, lane markings, traffic controls, nearby road users, and coarse ego-centric directional cues, providing auxiliary language supervision over topology-relevant scene semantics. Extensive experiments on OpenLane-V2 and OmniDrive-nuScenes demonstrate that TopoMind achieves state-of-the-art performance on both lane topology perception and topology-oriented understanding, while consistently surpassing perception-only topology models. These results demonstrate the effectiveness of jointly learning topology-oriented language supervision and lane topology perception within a unified representation.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Lane topology reasoning, large language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GoVmOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TopoMind combines lane topology prediction and driving-scene question answering in a single model. It uses a shared BEV–language representation and a Semantic-Geometric Query Refinement module (SGQR) to incorporate scene context into lane geometry and connectivity prediction. The authors also assemble approximately 1.19 million topology-oriented question–answer pairs for joint training. Experiments on OpenLane-V2 and OmniDrive-nuScenes evaluate topology prediction and scene understanding, showing improvements over the reported baselines. Ablation studies examine the contributions of language supervision, SGQR, and joint training.

Critical strengths
  1. The paper combines scene question answering with lane topology prediction, using shared representations and SGQR to incorporate road semantics into lane-query refinement. This design fits the needs of topology reasoning and offers a useful approach to applying language supervision to structured road perception.
  2. The method outperforms the reported baselines on both lane-segment and centerline tasks. For example, on the Subset-A lane-segment task, mAP improves from 34.4 to 41.6 and the topology score from 31.3 to 33.6, showing benefits for both geometric detection and connectivity prediction.
  3. Comparisons that remove text, omit SGQR, or directly concatenate queries help identify the sources of improvement. The results support the contributions of language supervision and the fusion strategy, strengthening the evidence for the method’s effectiveness.
Critical weaknesses
  1. Section 3.3 describes using ground-truth answer prefixes during training and generated text during inference, but it remains unclear which text states SGQR uses in the topology experiments and whether lane prediction requires answer generation first. This affects the information available to the topology branch, the correspondence between training and inference, and reproducibility. Until this is clarified, I have reservations about how to interpret the main results.

  2. The external topology baselines do not use LLMs or VLMs, while the cited Chameleon already uses VLMs for topology reasoning. Different settings may preclude a direct numerical comparison, but the paper needs a more concrete account of what it advances beyond this approach. The scene-understanding evaluation also needs to account for the protocols and results of HERMES and GaussianDWM when making state-of-the-art claims. The internal ablations support the design, but leave uncertainty about its advantage over existing approaches, which limits my assessment of the contribution.

  3. The method introduces a 2B language model but does not report end-to-end inference latency, throughput, or memory usage. While the accuracy gains over topology baselines without LLMs or VLMs are clear, the additional computational cost remains unknown. Reporting these measurements under matched hardware and input settings would help assess the accuracy–efficiency tradeoff.

Questions for authors
  1. What text is supplied during topology evaluation, and which text hidden states does SGQR use? Is lane prediction performed before answer generation, or does it depend on generated answers? Please clarify the corresponding training and inference settings and the exact procedure used for Tables 1 and 2. This directly affects my assessment of reproducibility and the sources of improvement.

  2. What is the main methodological advance over Chameleon? If a direct comparison is inappropriate, which differences in inputs or training settings prevent it? Also, does Table 6 use the same QA evaluation protocol as HERMES and GaussianDWM? If so, would the state-of-the-art claim still hold after including their results?

  3. Under the clarified inference procedure, what are the full model’s per-frame latency and peak memory usage? How much overhead does it introduce relative to the variant without the language branch under matched hardware and input settings? This would help assess the computational cost of the reported gains.

I am open to revising my rating if the authors’ response adequately addresses the key concerns above.

Reviewer_qgJkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new framework for both lane topology reasoning and understanding. Different from previous works, which focus on predicting the lane instances and the topology relationships between them, this paper additionally predicts scene and topology understanding results, which are represented by text.

A BEV tokenizer is used to extract BEV features from the input surround-view images, and a text tokenizer is used to tokenize the input text prompt. Then the BEV features (tokens) and text tokens are concatenated and fed into an LLM (Qwen3-VL-2B) to obtain refined tokens. The refined text tokens are then fed into an LM head to decode the text outputs. The refined BEV tokens are fed into a Semantic-Geometric Query Refinement (SGQR) module, together with the pooled text tokens, as key and value. The learned lane queries attend to the key and value by cross-attention. The refined lane queries and the BEV features are then fed into the lane decoder head to predict the lane instances and their topology relationships.

To supervise the understanding branch, the authors construct a topology-oriented VQA dataset of about 1.19M question–answer pairs by using Qwen3-VL-32B.

The authors conduct experiments on OpenLane-V2 and OmniDrive-nuScenes. The experiments show that their method achieves state-of-the-art lane topology performance and improved understanding metrics.

Critical strengths
  1. This paper proposes a new framework which unifies lane topology reasoning and language understanding for the first time. The BEV tokens, lane queries and text tokens exchange information with each other through the LLM and the Semantic-Geometric Query Refinement (SGQR) module. The experimental results also demonstrate that adding the understanding branch improves the performance. The ablation study shows that, compared with the full model, the performance drops by 3.0 mAP on OpenLane-V2 Subset-A when the text input is removed (LLM only (w/o text)).

  2. This paper constructs a topology-oriented VQA dataset of 1,188,301 question–answer pairs, covering topology-relevant scene semantics, including road structure, traffic controls, surrounding road users, and coarse ego-centric directional cues. If the dataset is released, it will be meaningful for the community.

Critical weaknesses
  1. The significance of the language understanding branch is unclear. At inference time, the text outputs of the understanding branch are not used by any downstream module, e.g., planning or decision making, which makes this branch more like an auxiliary supervision loss. And there is no explicit relationship between the input text prompt and the outputs of the lane decoder. For example, a question about the ego lane does not localize the ego lane and the downstream connections of the ego lane in the predicted lane graph. As a result, I consider that the experiments mainly show that the language supervision improves the perception performance, rather than the claimed "unified perception and understanding". If the authors consider the whole autonomous driving system, a prompt-guided lane decoder would be more meaningful for navigation, which makes the system able to benefit from the language input interface. I think this would be the real "unified perception and understanding".

  2. The evaluation of the understanding capability is weak. In the paper, the authors evaluate the understanding branch on OmniDrive-nuScenes with METEOR, ROUGE and CIDEr, which mainly measure the lexical similarity. So these metrics cannot reflect whether the answers correctly understand the lane topology and the scene. And the improvements of the unified model over the understanding-only model are marginal (+0.009 METEOR, +0.002 ROUGE and +0.010 CIDEr). The authors should build a validation set of the topology-oriented VQA on the OpenLane-V2 validation frames and design a method to evaluate the correctness of the answers.

Questions for authors
  1. What are the FLOPs, the number of parameters and the inference time of the proposed framework? Compared with other methods, the proposed method additionally adopts a 2B LLM. These numbers are important to judge whether the comparison is fair and whether the model can run in real time.

  2. Will the topology-oriented VQA dataset be open-sourced? If it is released, it will be meaningful for the community, which is one of the main strengths I consider.

Reviewer_nT73OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TopoMind, a framework for lane-topology reasoning and driving-scene understanding. It is built on a shared BEV and text LLM architecture, and introduces Semantic-Geometric Query Refinement (SGQR) to enhance lane geometry and connectivity prediction. The model is trained with about 1.19M topology-oriented QA pairs. Experiments on OpenLane-V2 and OmniDrive-nuScenes report perception gains and improvements in language understanding.

Critical strengths

Impressive gains from the framework. Relative to the authors’ own no-language baseline, the full model improves Subset-A by +4.7 mAP / +5.2 TOP_lsls (Table 3) and Subset-B by +8.3 L-OLS (Table 4). The gains hold across two data sources and two lane representations. It is a useful empirical signal that LLM-augmented, language-supervised training can improve lane-graph learning.

Meaningful unification of perception and understanding. The framework supports lane-graph prediction and topology-oriented VQA together. The joint training improves perception while maintaining understanding performance, though a more thorough comparison is needed.

Critical weaknesses

Insufficient positioning against prior work and understanding baselines.

The narrow joint lane-graph prediction and VQA formulation is one of the contributions, but many related works are insufficiently discussed or just omitted:

  • OmniDrive (CVPR'25) already couples lane-supervised 3D perception with an LLM; BEV-InMLLM (CVPR'24) integrates BEV features into an MLLM; MTA (CVPR'26) jointly optimizes BEV perception and captioning; and VLM-AD (CoRL'25) uses VLM-generated auxiliary supervision.
  • Chameleon (ICRA'25) already formulates lane-topology relations as VLM questions, while TopoAware-Bench (ICASSP'26) explicitly benchmarks topology-oriented VQA.

Table 6 reports only OmniDrive-BEV (CIDEr 0.595), omitting Omni-Q (0.686) and Omni-L (0.732). It also needs to include HERMES (ICCV'25), which reports CIDEr 0.741 versus TopoMind's 0.735 on OmniDrive-nuScenes. The authors should better distinguish their contribution from these prior approaches, clarify evaluation comparability, and verify the understanding SOTA claim.

Questions for authors
  1. Understanding baseline comparison. Please clarify whether Omni-L, Omni-Q, and HERMES use the same OmniDrive-nuScenes evaluation protocol as TopoMind. Their reported results appear competitive with, or better than, TopoMind's understanding scores.

  2. Sources of perception gains. The perception-only variants already substantially outperform LaneSegNet. What architectural or training differences account for this gap? Also, it would be helpful to clarify whether the gains from textual supervision are primarily driven by GT-assisted directional QA or broader scene-understanding supervision.

I would appreciate clarification of these points, and the responses may affect my final assessment.

Comments on the paper

1 comment

Dear Area Chair,

We responded point by point to all three reviews. Our new experiments test whether language helps predict lane graphs and whether learning to predict those graphs helps the model answer topology questions correctly.

1. Do directional questions account for the perception gains? (nT73, qgJk) We added a directional-QA-only run to the Subset-A ablations using our rebuilt LaneSegNet baseline.

Training variantmAPAP_lsAP_pedTOP_lsls
Perception only36.935.837.928.4
LLM without text38.637.839.431.3
GT-assisted directional QA only38.539.138.031.9
Without SGQR39.639.939.132.9
Full TopoMind41.642.141.133.6

Broader scene QA adds 3.1 mAP and 1.7 TOP over directional QA alone. Removing semantic-geometric query refinement (SGQR) lowers mAP/TOP by 2.0/0.7 relative to the full model. Appendix A clarifies how language enters the decoder in Tables 1 and 2. Training uses reference answers with teacher forcing. At inference, the LLM generates answers from questions and BEV tokens, then encodes the dialogue with the BEV tokens. Lane queries attend to the projected, pooled text states and recovered BEV features before predicting geometry and connections.

2. Are topology answers more accurate? (qgJk) On Subset-B validation frames, we scored ego-lane direction by accuracy and connected downstream directions by mean per-frame set F1, which credits partial matches.

ModelEgo direction accuracyDownstream direction F1
Understanding only83.88%62.75%
Unified87.53%66.56%

Joint training gains 3.65 points in accuracy and 3.81 in F1 on questions with verifiable answers. It also improves Subset-B centerline L-OLS from 46.3 to 54.6 over perception alone (Table 4).

3. What does TopoMind add over Chameleon? (GoVm) We added Chameleon’s published 3-shot result. It infers topology from lanes predicted by a separate detector with an SD map.

OpenLane-V2 methodSD mapLane detectionTOP_lsls
LaneSegNetNo32.3025.40
LaneSegNet + SD map + ChameleonYes38.3531.65
TopoMindNo42.1033.60

TopoMind jointly predicts lanes and connections from cameras without an SD map, exceeding this configuration by 3.75 points in detection and 1.95 in topology.

4. How does understanding compare across model sizes and modalities? (nT73, GoVm) We expanded the OmniDrive-nuScenes comparison beyond OmniDrive-BEV, the closest architectural baseline because both models pass BEV features to the LLM.

MethodLLMModalityMETEORROUGECIDEr
LLaVA-OV7BC→T—0.2210.284
OmniDrive-BEV7BC→T0.3560.2780.595
ORION7BC→T0.3540.3060.635
Omni-Q7BC→T0.3800.3260.686
Omni-L7BC→T0.3760.3210.732
GaussianDWM8BC+G→T+V0.3950.3410.712
HERMES1.8BC→T+L0.3840.3270.741
TopoMind1.7BC→T0.3800.3230.735

C denotes cameras, T text, L LiDAR point clouds, G reconstructed 3D Gaussians and V RGB-D video. With a 1.7B LLM, TopoMind exceeds all three 7B OmniDrive variants and 8B GaussianDWM on CIDEr. Without point-cloud supervision, it comes within 0.006 CIDEr of LiDAR-supervised HERMES.

5. What does the language branch cost? (qgJk, GoVm) Section 4.5 compares full answer generation and lane prediction on Subset-B with perception alone, and reports HERMES on the same A100 hardware.

Model/backendTotal paramsTFLOPs/frameMean s/framePeak GiB
Perception only64.74M0.6600.12910.81
TopoMind/HF2.35B77.33934.84514.62
TopoMind/vLLM2.35B16.8741.37925.74
HERMES/HF2.38B155.25217.68937.87

vLLM speeds up the same TopoMind model 25.3× over HF. TopoMind/vLLM takes 1.379 s/frame versus 17.689 for HERMES/HF and uses 12.13 GiB less peak memory. We will release the training and evaluation code, topology VQA dataset and generation engine.

All three reviews now carry a rating of 3 (Weak acceptance). GoVm raised their rating from 2 to 3 after our response. qgJk confirmed that most concerns had been addressed and maintained their positive rating. nT73 thanked us for the new experiment and requested two further revisions. We revised the abstract to claim state-of-the-art lane perception and competitive scene understanding, explained our Subset-B implementation, and added our perception-only LaneSegNet† baseline to Tables 1 and 2. With the same perception code and preprocessing, full TopoMind gains 4.7 mAP and 5.2 TOP_lsls on Subset-A lane segments and 6.7/8.3 L-OLS on Subsets A/B centerlines.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.