acceptodds
Under review as a conference paper at ICLR 2027

SD-LLaVA: Dual-Stream Vision-Language Reasoning with Structured Semantic Priors for BEV Understanding

Abstract

Large vision-language models (LVLMs) remain unreliable on Bird’s-Eye-View (BEV) semantic maps because their visual representations are primarily optimized for natural-image appearance, while BEV semantics are encoded through discrete labels, spatial structures, and geometric conventions. To address this representation mismatch, we propose SD-LLaVA, a dual-stream architecture that combines continuous visual features with explicit structured semantic priors for BEV-grounded reasoning. A SemanticPriorEncoder extracts class-aware spatial statistics from BEV maps and projects them into semantic tokens aligned with the language-model embedding space. These tokens are jointly integrated with CLIP patch representations, enabling the model to capture complementary visual context and explicit semantic structure. We further introduce a BEV question-answering protocol that separates factual grounding from open-ended driving-oriented reasoning, allowing systematic evaluation of both semantic understanding and response quality. Experiments on a BEVCar-derived benchmark show that SD-LLaVA achieves 90.97% semantic accuracy, substantially outperforming CLIP-only and semantic-only variants, while also improving subjective driving-oriented QA. These results demonstrate that bridging continuous visual representations with structured semantic priors provides an effective approach to grounding LVLMs in symbolic spatial environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Structured Semantic Priors, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_v71hOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SD-LlaVA, a dual-stream architecture that can help VLMs to understand BEV maps by explicilty extracting their semantics before passing them to lanugage model. It includes two streams: visual stream uses CLIP to convert BEV into visual tokens and semantic stream uses a handcrafted parser to split the image into four regions.

Critical strengths

Treating color as semantic labels is appropriate for maps whose colors have predefined meanings and the proposed semantic stream adds only four tokens, which is compact and interpretable.

Critical weaknesses
  1. The number of baselines are too limited. This idea of enhancing the understanding of VLM for BEV is not novel as it has been studied by [1] [2]. Thus, the author should include more baselines as comparison for the performance.
  2. The experiment setting is unclear. The author does not mention the dataset size, train/test partitioning, etc, which further obscures the contribution of this paper.

[1] Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K. Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K. Madhava Krishna. "Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving"

[2] Qingyao Xu, Siheng Chen, Guang Chen, Yanfeng Wang, Ya Zhang. "ChatBEV: A Visual Language Model that Understands BEV Maps"

Questions for authors

Similar as weaknesses.

Reviewer_J9dCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SD-LLaVA adds four semantic tokens to LLaVA for QA on color-coded BEV maps. CLIP encodes the map as 576 tokens; a 9-class palette parser counts vehicles and class presence in four diagonal regions and projects them into the four tokens. Objective labels come from the same masks. Subjective questions ask for a road description and 5–10s driving advice, scored by GPT. With LLaVA-v1.6-13B, semantic accuracy is 29.59%, 73.42%, and 90.97% for CLIP-only, semantic-only, and the full model; subjective scores are 45.30, 46.86, and 69.96.

Critical strengths

None

Critical weaknesses

The paper motivates SD-LLaVA as support for driving understanding and short-horizon decisions, but never places the model in that loop. A BEV semantic map is an intermediate perception product: it is estimated from sensors and may be wrong, and a driving action is useful only if it is executed and checked against the next observation. SD-LLaVA instead takes a finished color map as input and returns text. Factual labels are derived from the same map, so objective accuracy measures whether the model restates the map. The stated difficulty, CLIP misreading symbolic colors, is therefore not the decision problem the introduction invokes, and the reported gains do not show that the interface changes what a vehicle would do.

Questions for authors
  1. Are the input maps ground-truth color renderings, or BEV maps predicted by BEVCar or another perception model? If the maps contain class errors, missing objects, or bad boundaries, which input could reveal that the map is wrong?

  2. Are the objective labels computed from the same palette statistics that form (V)? If so, what part of semantic accuracy is left once those statistics are already given to the model?

Reviewer_TvB2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SD-LLaVA for BEV semantic map QA. Besides the standard CLIP visual tokens, the authors add a SemanticPriorEncoder that splits the color-coded BEV map into four diagonal regions, uses RGB color matching and connected components to get regional object counts and class presence, and projects this 4x9 matrix into 4 tokens for the LLM. They test it on a BEVCar-based QA dataset with objective and subjective questions.

Critical strengths

None

Critical weaknesses
  1. The paper is clearly an unfinished draft. Section 3 is literally just a section title with nothing under it, while the tables are oddly placed in Section 2 and re-plotted as bar charts in Section 4. Basic details like dataset size, splits, training settings, and QA generation rules are completely missing. The text also refers to a non-existent appendix, and there are no comparisons against existing BEV-VLM baselines (e.g., Talk2BEV, ChatBEV).

  2. The SemanticPriorEncoder is basically a rule-based script that leaks the answers into the input. It uses RGB color matching and connected components on clean masks to pre-compute regional vehicle counts and class presence, then feeds them to the LLM via a linear layer. The objective QA benchmark then tests the model on these exact same counts and presence statistics. As shown in Table 1, the Semantic-Only variant (which only sees these 4 tokens and no image) already gets 88.98% counting accuracy vs. 5.31% for CLIP-Only. Getting ~90% here just shows the LLM can read out pre-computed numbers, not visual reasoning.

Questions for authors

Given that Section 3 is missing and the evaluation has target leakage issues, I don't think a rebuttal can fix the paper in this cycle. I encourage the authors to complete the experimental section, add proper baselines, and rethink the rule-based counting module before resubmitting to a future conference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.