acceptodds
Under review as a conference paper at ICLR 2027

From Scene Graphs to Answers: Selective Neuro-Symbolic Reasoning for Autonomous Driving

Abstract

Autonomous-driving question answering requires reasoning over structured scene information, yet existing vision-language approaches largely delegate heterogeneous reasoning operations to a single neural inference process. We argue that this uniform strategy overlooks a fundamental distinction: some queries admit exact symbolic solutions, while others require semantic interpretation. We introduce a query-adaptive neuro-symbolic reasoning framework that explicitly allocates computation according to the nature of the query. At its core is a hierarchical Spatiotemporal Scene Graph (STSG) that separates persistent object identities from frame-specific states and represents spatial relations and temporal transitions as explicit directed structures. Given a query, a symbolic executor first attempts to resolve it through exact graph operations; only when symbolic execution abstains is an LLM invoked for semantic reasoning. For these unresolved queries, query-conditioned graph retrieval and evidence filtering preserve relation direction, temporal locality, and object semantics, providing the LLM with compact and verified task-relevant evidence. This design shifts the role of the LLM from a universal reasoning engine to a targeted semantic reasoner, while allowing deterministic computation to be handled exactly and efficiently. We evaluate the framework on 5,916 NuScenes-QA questions across all ten scenes of nuScenes v1.0-mini under an oracle-perception setting. The complete system achieves overall accuracy with GPT-5.4-mini, improving over the corresponding LLM-only configuration by 5.48 percentage points; with DeepSeek-V4-Flash, the improvement reaches 6.64 points. The largest gains occur on counting questions, with improvements of 10.20 and 12.61 points, respectively. These results demonstrate that selective allocation of reasoning can improve both accuracy and inference efficiency, providing a principled alternative to treating the LLM as a universal executor for structured reasoning in autonomous driving.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-symbolic reasoning, Spatiotemporal scene graphs, RAG, Question answering, Autonomous driving, Spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wmoFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a symbolic-first framework for driving-scene question answering. It constructs a spatiotemporal scene graph, answers supported queries through deterministic operations, and delegates unresolved queries to a retrieval augmented LLM. Experiments evaluate 5,916 frame-specific questions from ten nuScenes-mini scenes using ground-truth annotations.

Critical strengths
  1. Selective symbolic execution improves accuracy by 5.48–6.64 percentage points and reduces QA runtime by 30.3–43.5% across two LLM backends, demonstrating useful gains over the LLM-only configuration.

  2. Explicit graph facts and deterministic operations make the answering process inspectable and potentially useful for offline driving-data queries and analysis.

Critical weaknesses
  1. The methodological novelty is limited. The framework combines established scene-graph representations, symbolic execution, and retrieval-augmented LLM reasoning. The benefit of its spatiotemporal structure is not demonstrated, since evaluation uses only frame-specific questions and lacks a frame-level graph baseline.

  2. Practical applicability remains insufficiently validated. Experiments use ground-truth observations from only ten scenes, without predicted perception or robustness tests. The results therefore establish offline QA performance under ideal inputs, but leave reliability under missing objects, incorrect attributes, and tracking errors unclear.

Questions for authors
  1. What benefits does the hierarchical STSG provide over independent frame-level graphs on the current benchmark?

  2. What is the intended deployment scenario, and how would predicted perception outputs be integrated into the system?

  3. How does performance change under missing detections, incorrect attributes, or identity-switch errors?

Reviewer_gni8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a selective neuro-symbolic framework for autonomous-driving question answering. It constructs a hierarchical spatiotemporal scene graph from ground-truth nuScenes annotations, representing persistent objects, frame-specific states, spatial relations, and temporal transitions. Given a question, a template-aware symbolic executor first attempts to answer it using deterministic graph operations; unsupported or ambiguous questions are handled by an LLM using retrieved and filtered graph evidence. Experiments on 5916 NuScenes-QA questions from nuScenes-mini report improved answer accuracy and reduced LLM inference time compared with an LLM-only pipeline.

Critical strengths
  1. The distinction between deterministic graph operations and semantic LLM reasoning is intuitive, and the abstention-based routing design is easy to understand and inspect.

  2. The paper is clearly organized and reports per-question-type results, comparisons across two LLM backends, and runtime measurements, making the behavior of different components relatively transparent.

Critical weaknesses
  1. Fundamentally circular oracle evaluation. NuScenes-QA is itself programmatically generated from nuScenes 3D annotations, scene graphs, and manually designed question templates. The proposed method reconstructs a graph from the same ground-truth 3D boxes, categories, instance identities, attributes, and ego poses, and then uses a template-aware symbolic executor to recover the answers. This bypasses perception entirely and largely reproduces the benchmark-generation logic rather than demonstrating visual scene understanding.

  2. Limited relevance to autonomous driving. The system processes neither images nor point clouds and does not evaluate planning, trajectory prediction, control, or safety. Both the symbolic and LLM branches operate on oracle structured facts. Therefore, the experiments support only a narrow claim about structured QA, not the broader claim of reasoning for autonomous driving.

  3. The spatiotemporal contribution is not validated. Evaluation uses only 5916 frame-specific questions from 404 frames in the ten nuScenes-mini scenes. None of the questions requires explicit cross-frame reasoning, so the persistent identities and temporal-transition edges—the main distinction of the proposed STSG—are not experimentally shown to be useful. Generalization to unseen scenes or question templates is also not evaluated.

  4. Limited technical insight and incomplete efficiency evaluation. Using exact graph operations for existence and counting before invoking an LLM is straightforward, and the large improvements are expected when ground-truth instance identities are available. Moreover, the runtime analysis excludes graph construction and indexing and is affected by remote API latency, while the complete system still requires approximately 15–23 seconds per question. Thus, the claimed efficiency benefit is only partially established.

Questions for authors

See weaknesses.

Reviewer_LvgbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Spatiotemporal Scene Graph (STSG), which represents objects, their states, spatial relations, and temporal state changes in autonomous-driving scenes. It uses a symbolic-first question-answering pipeline: a rule-based symbolic solver answers questions it considers supported, while questions it cannot answer or considers ambiguous are handled using LightRAG retrieval and an LLM. The authors evaluate the system on 5,916 NuScenes-QA questions from 10 nuScenes v1.0-mini scenes. With GPT-5.4-mini and DeepSeek-V4-Flash, the full pipeline reportedly improves overall accuracy over the corresponding LLM-only setup by 5.48 and 6.64 percentage points, respectively, with larger gains on counting questions. However, the evaluation uses ground-truth 3D object information, and the reported runtime covers the QA stage only, excluding scene-graph construction and index preparation.

Critical strengths
  1. An interpretable symbolic-first routing design.
    The system selects between a symbolic solver and an LLM based on whether a question appears answerable through rules. This is a practical direction that could reduce LLM calls for suitable questions and provide structured grounds for answers.

  2. Reported improvements over LLM-only systems with two backends.
    The full pipeline reports higher overall accuracy than the LLM-only setup for both GPT and DeepSeek, while also reducing QA-stage runtime. These results suggest potential practical value, although the current experiments do not fully isolate the cause of the gains.

Critical weaknesses
  1. The evaluation is limited, and the claimed contribution of temporal reasoning is not demonstrated. \ The evaluation covers only 10 nuScenes mini scenes and relies on ground-truth 3D object information. This makes it difficult to assess performance under realistic perception errors or across a broader range of scenes. In addition, the evaluated questions are frame-specific, so the experiments do not establish whether the temporal links in the STSG contribute to performance. The paper’s emphasis on spatiotemporal reasoning is therefore not supported by the current evaluation.

  2. The comparisons do not sufficiently isolate the method’s effects, and key reliability questions remain unanswered.
    The number of baselines is limited, and the Symbolic-only configuration is not a strict component ablation. Comparison questions are routed to the neural fallback in both the full and LLM-only settings, yet their reported accuracy changes from 66.44 to 68.67 for DeepSeek and from 76.21 to 74.08 for GPT. The paper does not explain these differences or report repeated runs and uncertainty, making it difficult to judge the stability of the results. The evaluation also does not report how often symbolic and LLM answers disagree, or whether the LLM can detect or correct cases where the symbolic solver returns an incorrect answer. As a result, the source of the reported gains and the reliability of the routing strategy remain unclear.

Questions for authors
  1. Comparison questions appear to use the neural fallback in both the full and LLM-only settings. What accounts for the changes in their accuracy? Were the inputs, prompts, and execution conditions identical across settings, and can you report results across repeated runs?

  2. Did you run the symbolic solver and the LLM on the same questions to measure answer disagreement? In particular, how often does the symbolic solver return an incorrect answer rather than abstaining, and how does the system handle such cases?

  3. Do you have results on questions that require information across multiple frames, or an ablation showing whether the temporal links in the STSG improve performance? Such results would help establish the contribution of the temporal representation to the paper’s central claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.