acceptodds
Under review as a conference paper at ICLR 2027

Intent2Query: From Natural Language Intent to Executable Spatiotemporal Queries for Multi-Agent Trajectory Retrieval

Abstract

Retrieving semantically meaningful spatiotemporal trajectories from large-scale driving data is essential for analyzing complex traffic behaviors and improving autonomous systems, yet remains challenging. Multi-agent interactions evolve over continuous space and time, making retrieval inherently difficult. Meanwhile, existing systems lack both formally verifiable query constraints and an accessible interface aligned with natural language intent, which is typically more intuitive and flexible than example-driven similarity mining. Moreover, such intent is frequently ambiguous and requires semantic grounding together with quantitative, verifiable evidence to ensure correctness. We introduce Intent2Query, a framework that translates natural language (NL) intent into a formally defined spatiotemporal query language and executes retrieval programs under Boolean and quantitative semantics. The proposed domain-specific language (DSL) is grounded in signal temporal logic (STL), enabling precise specification of multi-agent behaviors and spatiotemporal reasoning with quantitative robustness. To ensure reliability, parsing is constrained by schema and grammar rules, followed by deterministic validation, guaranteeing executable queries. We further construct a human-annotated scene-level benchmark over intersection LiDAR data to evaluate both parsing fidelity and retrieval performance. Results show that Intent2Query outperforms all ten baselines on all three datasets and at every difficulty level, improving F1 from 0.496 to 0.682 on our benchmark with non-overlapping confidence intervals and nearly doubling the precision of an untyped temporal-logic parser at comparable recall (0.636 v.s. 0.330), highlighting the necessity of typed formal semantic grounding for reliable spatiotemporal retrieval.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, Domain Specific Language, Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B26POpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Intent2Query, which translates natural-language descriptions of traffic interactions into a typed query language grounded in signal temporal logic (STL), then executes those queries over multi-agent trajectories. It reports improved retrieval performance across three datasets.

Critical strengths

The problem is well motivated. Finding specific interactions in large trajectory datasets is useful for autonomous-driving research, and a natural-language interface could reduce the effort of writing search scripts. Typed participants and map-aware predicates are sensible design choices, with substantial gains reported on LADDMS.

Critical weaknesses

The retrieval ground truth needs clearer documentation. Appendix F describes human relevance judgments, but does not specify annotation coverage, how partial and uncertain labels enter the retrieval metrics, or how disagreements were handled. The source of binary relevance labels for NGSIM and INTERACTION is also unclear. These details are essential for interpreting the reported precision and recall.

The baselines are insufficiently specified. Section 5.3 describes the method families, but omits important details such as the embedding and reranking models, scene representations, example selection, and tuning procedures. Without these details, it is difficult to assess baseline strength or reproduce the comparisons.

The technical contribution needs sharper positioning. STL is a reasonable foundation, and the paper acknowledges that quantitative robustness mainly benefits ranking. However, the distinction between the proposed DSL and existing formal trajectory-query approaches remains underdeveloped. The paper should clarify which capabilities are new and which come from integrating established techniques. In general, I'm against abusing STL if the core capabilities of STL are not central to the method.

Questions for authors

How were retrieval labels constructed for each dataset, and what fraction of query–scene pairs received independent human annotation?

Which exact implementations and configurations were used for the baselines?

2 comments
AuthorsOpenReview ↗

Response to Reviewer B26P

We thank the reviewer for the careful and constructive review.

Summary. We have uploaded a revised manuscript clarifying retrieval labels, documenting all 12 baselines (Appendix H), adding stronger comparisons (a 14B LLM judge, RefAV, alternative text embed models and fine-tuned embedder), and sharpening our technical positioning. Below, we address annotation reliability (W1), baseline reproducibility (W2), and the novelty and central role of STL (W3).

W1, Q1 (Retrieval labels and annotation). LADDMS labels were initialized using an independent executable reference labeler and subsequently verified through human annotation. Two domain experts independently labeled query–scene pairs based on the reference results, with disagreements resolved by a third expert. NGSIM and INTERACTION labels were generated using the same reference labeler.

Only LADDMS supports ranking evaluation with human-annotated graded relevance (exact/partial/negative/unsure). For nDCG@k, exact/partial/negative receive gains of 3/1/0, while unsure pairs are excluded; precision, recall, and F1 use binary relevance.

W2, Q2 (Baseline implementations and configurations). Appendix H now documents all 12 baselines, including input representations (z-scored scene features, trajectory signals, and scene captions), model identities and prompts (Qwen2.5-family LLMs and open-source Hugging Face embedders; Table 16), and evaluation protocols. To ensure fair comparisons:

  • Leakage prevention: for every score-based baseline (Few-Shot, the embedders, the rerankers and the fine-tuned embedder), intents are split into two folds grouped by the gold program's predicates, operators and agent types, so paraphrases never cross folds; cutoffs, and for the fine-tuned model its training labels, come only from the opposite fold.
  • Example selection: Few-Shot and QBE baselines receive their specified numbers of labeled examples (k relevant or 10 positive and 10 negative), sampled five times and excluded from evaluation.
  • Faithful implementation: RefAV follows its own prompting and execution pipeline (Appendix H.6).

Under this protocol, Intent2Query significantly outperforms all baselines on LADDMS, including:

Method (LADDMS)F1ΔF1 vs. Intent2Query
Intent2Query0.738—
14B LLM judge0.573+0.165
RefAV0.569+0.169
Fine-tuned bge-large0.501+0.237
QBE-DT (20 labeled scenes)0.493+0.245

W3 (Technical novelty and the role of STL). We clarify three contributions: (1) typed multi-agent grounding, with distinct, type-constrained agent variables shared across signal, relational, and map predicates; (2) formal retrieval semantics, maximizing scene-level robustness over valid bindings and observation times; and (3) validated natural-language querying, with deterministic program validation.

We do not claim novelty in bounded discrete-time STL robustness or semantic parsing to temporal logic; rather, we integrate them into a unified retrieval framework. Unlike Quivr, which synthesizes queries from labeled examples, Intent2Query requires none: our QBE-STL baseline achieves F1 = 0.453 with 20 labeled scenes per intent, versus 0.738 for Intent2Query. Scenic-based approaches require expert-written programs, while STREL and Traffic Scenario Logic primarily target monitoring or scenario generation.

STL is central, not auxiliary: bounded temporal operators support ordering and metric-time constraints required by 91/300 intents, while quantitative robustness enables principled ranking and adjustable retrieval thresholds. Robustness ranking improves nDCG@10 by +0.050 (95% CI [+0.028, +0.073]) over Boolean matching, and removing temporal operators reduces F1 by 0.057 (Table 12). Adjusting the robustness threshold also enables precision–recall trade-offs without retraining.

In summary, STL provides the established temporal and quantitative foundation; our contribution lies in extending it with typed multi-agent grounding, formally defined retrieval semantics, and validated natural-language querying.

We thank the reviewer again for the valuable feedback, which has strengthened the manuscript. We hope these revisions address the concerns and would appreciate reconsideration of the rating.

Dear Reviewer B26P,

We sincerely thank you for your thoughtful review and for recognizing the importance of the problem and the value of typed, map-aware querying. Your feedback has helped us improve the manuscript, particularly by clarifying the annotation procedure, providing detailed baseline configurations, and sharpening the technical novelty and central role of STL.

We kindly invite you to review our detailed rebuttal and revised manuscript before the discussion period closes. We hope these revisions address your concerns about technical contributions, and we would greatly appreciate your reconsideration of the evaluation. Please let us know if further clarification would be helpful.

Best regards,

Authors of Submission 35256

Reviewer_9KHZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Intent2Query, a multi-agent trajectory retrieval framework that converts natural language intents into executable spatiotemporal queries. Specifically, they design a typed Domain-Specific Language based on Signal Temporal Logic. During the parsing stage, an LLM combined with grammar-constrained decoding generates the DSL program, which then undergoes canonicalization, deterministic repair, and validation to ensure executability. During the execution stage, the framework returns matched scenarios ranked by their degree of satisfaction through macro expansion, type-constrained variable binding, and robustness computation. Additionally, the authors construct a human-annotated benchmark comprising 300 queries.

Critical strengths
  • The problem formulation is valuable; such formalized programs are highly useful for data mining and safety verification in autonomous driving.

  • The DSL design is clearly defined. Typed variables, map predicates, temporal operators, and quantitative robustness all have rigorous formal definitions.

  • The experimental evaluation covers a broad scope, comparing against 10 baselines and testing across three datasets.

Critical weaknesses
  • The novelty of Intent2Query is limited, as there is already a substantial body of similar work. For instance, RefAV [1] and the series of works from the Argoverse 2 [2] scenario mining challenge take a very similar approach: they define a set of hand-crafted atomic function APIs (e.g., turning, accelerating, lane changing) and prompt an LLM to synthesize a program that composes these atomic actions to match natural language descriptions. Meanwhile, works such as NL2TL [3] have already investigated translating natural language into STL. In other words, the individual components of the paper—including natural language to temporal logic translation, grammar-constrained decoding, and STL robustness—are all covered by existing work, making the primary contribution an incremental combination.

  • The scale of the benchmark dataset is small. It consists of only 300 queries, 270 of which are generated by an LLM and enumerated according to the grammar coverage of the DSL. This implies that all queries are inherently expressible within this DSL, which favors DSL-based methods and fails to evaluate how the DSL handles intents that fall outside its expressive capacity.

  • Both retrieval performance and efficiency are limited: on the INTERACTION dataset, the F1 score is only 0.388 overall and merely 0.250 on hard queries, outperforming baselines by only a narrow margin. Moreover, across 940 scenarios, each query takes an average of 69 seconds—with the 95th percentile reaching 300 seconds and experiencing timeouts—and the authors also acknowledge that the current system is unsuitable for million-scale scenario retrieval.

[1] Davidson, Cainan, Deva Ramanan, and Neehar Peri. "Refav: Towards planning-centric scenario mining." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Wilson, Benjamin, et al. "Argoverse 2: Next generation datasets for self-driving perception and forecasting." arXiv preprint arXiv:2301.00493 (2023).

[3] Chen, Yongchao, et al. "Nl2tl: Transforming natural languages to temporal logics using large language models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Questions for authors
  1. What is the core distinction between Intent2Query and methods like RefAV that also use LLMs to generate executable programs for scenario mining?

  2. Since the parsing evaluation is based on only 30 queries, have the authors explored evaluating on a larger scale? Additionally, what model size was used for retrieval in the main experiments, and was grammar-constrained decoding applied there?

  3. Why does automated parsing achieve a slightly higher F1 score than the human-written gold-standard programs? Does this indicate a discrepancy between the predicate thresholds/definitions and the human annotations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9KHZ

We thank the reviewer for the careful and detailed review.

Summary. We have uploaded a revised manuscript with stronger baselines (RefAV, a larger LLM judge, and improved embedding models), expanded parsing evaluation on all 300 intents (Table 2, Appendix I), and updated retrieval results. Below, we clarify our novelty (W1, Q1), benchmark coverage (W2), retrieval performance and efficiency (W3), parser configuration (Q2), and oracle evaluation (Q3).

W1, Q1 (Novelty; distinction from RefAV and NL2TL). We agree that the individual techniques are established; our contribution lies in the target representation and its formally grounded retrieval semantics. RefAV synthesizes Python programs composing sets of (track, timestamp) pairs, whereas Intent2Query generates typed, first-order queries with bounded discrete-time STL semantics. This provides three key capabilities: (1) metric temporal operators, required by 91/300 intents; (2) quantitative robustness for scene ranking and adjustable precision–recall trade-offs; and (3) deterministic validation before execution.

We implemented RefAV's pipeline using Qwen2.5-7B with 8 in-context examples, adapting its API to our datasets (15 retained functions and 4 additions). One program is generated per intent and reused across datasets. Intent2Query significantly outperforms RefAV on all three datasets: LADDMS (F1 0.738 vs. 0.569), NGSIM (0.966 vs. 0.834), and INTERACTION (0.392 vs. 0.329). Unlike Intent2Query, NL2TL translates language into formulas over abstract propositions without typed multi-agent grounding, trajectory execution, or quantitative scene ranking.

W2 (Benchmark scale and expressiveness). Our 300 intents are evaluated against every scene across three datasets; RefAV similarly evaluates 315 prompts generated from its atomic API. We acknowledge that our benchmark primarily measures fidelity on expressible intents and additionally examine representational fairness: RefAV can express 209/300 intents, while standard LTL can express 283/300. On the 209 intents expressible by all methods, Intent2Query still achieves higher F1 (0.768) than RefAV (0.631), QBE-DT, and NL2LTL-Rule.

The DSL is extensible: new behaviors can be supported through predicate registry and lexicon entries without retraining the parser or modifying the executor. Runtime scales linearly with corpus size; extrapolation suggests approximately 1 minute for one million scenes on a 192-core CPU under ideal parallelism, with candidate pruning and GPU acceleration offering further optimization opportunities.

W3 (INTERACTION performance and efficiency). INTERACTION contains only 51 scenes, and 75/200 intents have no relevant matches, limiting the maximum macro-F1 to 0.625. On the 118 intents with at least two relevant scenes, Intent2Query achieves F1 = 0.653, statistically comparable only to label-dependent or per-scene LLM baselines, while significantly outperforming the others.

The originally reported 69 s/query resulted from an x86 Python build running under Rosetta. Native execution reduces latency to 2.0 s median and 10.4 s mean per query, with only 4/300 timeouts (Appendix E), compared with 213 s/query for LLM judges. Recall@10 is low across methods because a median LADDMS query has 61 relevant scenes.

Q2 (Parsing evaluation and GCD). All main retrieval results use the unconstrained Qwen2.5-7B parser, the strongest parser in Table 2; GCD is evaluated separately. We expanded parsing evaluation from 30 to all 300 intents (Table 2, Appendix I). GCD improves Hard-query validity for 0.5B (0.00 → 0.48) and 1.5B (0.08 → 0.73), but reduces it for 7B (0.91 → 0.43), consistent with the earlier evaluation. Thus, GCD primarily benefits smaller models rather than consistently improving performance across model scales; we retain it as an option for settings where only smaller models are available.

Q3 (Oracle evaluation). The evaluation in Table 3 shows that oracle programs achieve higher overall F1 and consistently higher precision across difficulty levels. Parsed queries sometimes achieve higher recall because they are less restrictive: among 37 intents where parsing outperforms the oracle, 33 retrieve a strict superset of the oracle's matching scenes.

We thank the reviewer again for the valuable feedback, which has strengthened the manuscript. We hope these revisions address the concerns and would appreciate reconsideration of the rating.

Dear Reviewer 9KHZ,

We sincerely thank you for your detailed review and constructive suggestions. In response, we have substantially revised the manuscript, including new comparisons with RefAV and stronger baselines, expanded parsing evaluation, and clearer discussions of novelty, benchmark coverage, and efficiency.

As the discussion period approaches its close, we would greatly appreciate your consideration of our rebuttal and updated manuscript. We hope these additions address your key concerns and provide stronger evidence for our contributions. We would be grateful if you could reconsider your assessment, and we welcome any further questions.

Best regards,

Authors of Submission 35256

AuthorsOpenReview ↗

Follow up to Reviewer 9KHZ

Dear Reviewer 9KHZ,

We appreciate your time and thoughtful review. As the discussion period is nearing its end, we would be grateful if you could share any remaining concerns regarding our rebuttal and revised manuscript.

We have made substantial efforts to address your key concerns, particularly through the new RefAV comparison, expanded parsing evaluation, and additional analyses of benchmark coverage and efficiency. We hope these results provide stronger evidence for our contributions and would greatly appreciate your feedback and reconsideration of your assessment.

Thank you again for your time and consideration.

Best regards,

Authors of Submission 35256

Reviewer_7VXwOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper proposes a neurosymbolic framework for grounding free-form natural language queries into formally verifiable trajectory queries. The method uses an LLM to map natural language queries to predefined predicates, operators, and atomic expressions, enabling formal verification of trajectory alignment along various dimensions. The authors demonstrate higher retrieval precision across query difficulty levels on multiple public benchmarks, although recall is generally lower. Additional details are needed regarding the evaluation methodology, particularly the prompts provided to embedding-based baselines.

Critical strengths

Strengths

  • Well-written paper with a clearly defined problem formulation.
  • Strong retrieval precision against several relevant baselines on multiple public benchmarks across varying query difficulty levels.
  • Thorough ablation studies investigating the impact of different design choices, runtime latency, and scalability of human data annotation.
  • Interesting and relevant approach, particularly for well-defined domains such as autonomous driving.
Critical weaknesses

Weaknesses

Major:

  • More details are needed regarding the prompts provided to embedding-based baselines. Additionally, the authors should justify the lack of comparison with fine-tuned embedding models, which can achieve strong retrieval performance on public benchmarks.
  • The approach requires a predefined, domain-specific DSL, which limits its ability to handle arbitrary natural language queries. As such, claims about supporting flexible and expressive natural language queries seem overstated. The authors should clarify how the system handles behaviors or actions that cannot be expressed using the predefined predicates and operators.
  • Missing comparisons against stronger LLM baselines, including closed-source models and larger open-source LLMs with stronger reasoning capabilities. These would provide a more representative comparison against state-of-the-art approaches.

Minor:

  • The claim that the proposed approach outperforms all other methods across query difficulty levels is overstated. Recall appears consistently lower on medium and hard queries, suggesting that the method misses relevant spatiotemporal trajectories despite achieving higher precision.
  • More details on how disagreements between human annotators were resolved would be useful, particularly for difficult queries where the decision boundaries may be ambiguous.
Questions for authors

See major weaknesses in the critical weaknesses section for key points and clarifications to address.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7VXw

We thank the reviewer for the careful and constructive review.

Summary. We have uploaded a revised manuscript with stronger baselines (a 14B LLM judge, RefAV, and a fine-tuned embedder), detailed baseline implementations and prompts (Appendix H), and additional ranking evaluations (Table 6). The new results demonstrate advantages over stronger baselines (Major 1, 3), clarify the DSL's extensibility (Major 2), and address retrieval performance across difficulty levels (Minor 1).

Major 1 (Embedding prompts and fine-tuning). Embedders use model-specific retrieval templates over deterministic whole-scene captions (Appendix H). Additional evaluations with CoSENT-fine-tuned bge-large-en-v1.5 and other embedding models show that even with stronger text embedding or fine-tuning, relevant baselines still underperform Intent2Query across three datasets (Table 17), particularly on LADDMS, the most diverse dataset, despite Intent2Query requiring no labeled training data.

Method (F1 · ΔF1)LADDMS (ours 0.738)NGSIM (ours 0.966)INTERACTION (ours 0.392)
Fine-tuned bge-large0.501 · +0.2360.947 · +0.0180.365 · +0.027
14B LLM judge0.573 · +0.1650.912 · +0.0530.377 · +0.014

Major 3 (Stronger LLM baselines). We added new baselines of Qwen2.5-14B judge using the same prompt, captions, and decision rule as the 7B model, and RefAV, which generates programs over atomic trajectory functions. Intent2Query significantly outperforms both on LADDMS and NGSIM. On INTERACTION, it significantly outperforms RefAV (+0.062, 95% CI [+0.020, +0.106]) and performs comparably to the 14B judge, even with its threshold tuned on labels, while our method does not need. RefAV achieves F1 = 0.569/0.834/0.329 versus our 0.738/0.966/0.392 across the three datasets. Moreover, LLM judges require inference for every query–scene pair (213 s/query for 7B), whereas Intent2Query completes queries in seconds. Closed-source evaluation was constrained by cost: 282,000 calls on LADDMS alone would exceed an estimated $3,000 for one frontier model.

Major 2 (DSL expressiveness and limitations). We agree that the scope should be explicit. Intents involving unsupported behaviors are rejected by the validator rather than silently executed. Intent2Query is extensible: its STL semantics, parser, validator, and executor are independent of the predicate registry, which operates over available dataset signals. New behaviors can be added through registry and lexicon entries without retraining, while remaining composable with existing operators. Predicates requiring unavailable information (e.g., traffic-light states or lane graphs) are inherently dataset-dependent, a limitation shared by RefAV. Automating predicate extension is a direction for future work.

Minor 1 (Difficulty claims and recall). We have refined our claim: Intent2Query achieves the highest F1 in 7/9 dataset–difficulty combinations. The two exceptions are Easy queries, where RefAV leads by only 0.018 and 0.025 (not statistically significant). On Medium and Hard queries, Intent2Query achieves the highest F1 across all datasets, with significant gains over all 12 baselines on LADDMS and NGSIM.

Recall is 0.958/0.915/0.773 for Easy/Medium/Hard queries (Table 3). Recall@10 is limited by the large number of relevant scenes (median 61 per query), but recall reaches 0.60 at depth 100 and 0.853 over full retrieval (Table 6). Lower Hard-query recall reflects a precision–recall trade-off: 84% of missed relevant scenes fail a discrete condition, and admitting them raises recall to 0.988 but approximately halves precision. We retain the strict threshold ρ > 0 to prioritize reliable matches, as higher-recall baselines often over-retrieve (returning all scenes yields F1 = 0.209). When higher recall is needed, relaxing the threshold to ρ > −2 increases recall from 0.846 to 0.985 with only 17 additional scenes per query, without retraining.

Minor 2 (Annotation disagreements). LADDMS labels were initialized using an executable reference labeler and independently verified by two domain experts, who agreed on 87.1% of annotated pairs. A third expert resolved disagreements. NGSIM and INTERACTION labels were generated using the same reference labeler.

We thank the reviewer again for the valuable feedback, which has strengthened the manuscript. We hope these revisions address the concerns and would appreciate reconsideration of the rating.

Dear Reviewer 7VXw:

We sincerely thank you for your feedback and for recognizing the strengths of our work, particularly its clear motivation, strong retrieval precision, and thorough evaluation. Your suggestions have helped us further strengthen the paper through additional baselines and clearer methodological details.

We would greatly appreciate it if you could review our rebuttal and revised manuscript. We hope the revisions have addressed your remaining concerns and further strengthened your confidence in our work. Please let us know if any additional clarification would be helpful.

Best regards,

Authors of Submission 35256

Reviewer_7VXwOpenReview ↗

Reviewer Response

The authors addresses my primary concerns regarding baseline comparisons, potential generalization overclaims, and limitations. It is promising that their approach outperforms finetuned LLM baselines on several datasets despite not requiring any training. It is also commendable that their method is significantly faster and more cost effective than closed-source LLM judges as well. Despite the cost limitations, it would be helpful in the final manuscript to provide at least a small-scale comparison with a frontier LLM since open-scale LLMs are still less capable even after finetuning in terms of generalization. It would be helpful for the authors to mention how to overcome generalization to open-set spatiotemporal queries. I will revise my score accordingly given these clarifications.

AuthorsOpenReview ↗

Thank you!

Dear Reviewer 7VXw,

We sincerely thank you for your thoughtful feedback, recognition of our improvements, and willingness to raise your score. We greatly appreciate your support.

We agree that frontier LLM comparisons and open-set generalization are important directions. We will include the value of a small-scale frontier LLM comparison in the final manuscript. For open-set query generalization concerns, new wording and new combinations of supported behaviors already need no new vocabulary: The LLM parses natural-language intents into DSL formulas using in-context examples (Table 2), and the same predicates, thresholds and parser run unchanged on NGSIM and INTERACTION (Table 4). For behaviors outside the vocabulary, we will add a discussion of two extensions:

  1. LLM-proposed STL predicates. An LLM proposes the definition of a new behavior as an STL formula over existing signals; the validator type-checks it and a user reviews it before it enters the predicate registry, without retraining.
  2. Hybrid retrieval for partially inexpressible intents. Formal execution of the expressible part retrieves candidate scenes and ranks them by STL robustness; a neural scorer evaluates only the part the DSL cannot express, keeping or demoting candidates accordingly.

Thank you again for your constructive suggestions, which have strengthened our work.

Best regards,

Authors of Submission 35256

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Area Chair

Dear Area Chair,

Thank you for handling our submission. We believe our responses have sufficiently addressed the reviewers’ concerns through substantial new results and clarifications. We have uploaded a revised manuscript that includes stronger baselines , detailed baseline implementations and prompts (App. H), and additional ranking evaluations (Table 6).

We also thank Reviewer 7VXw for recognizing our work as an “interesting and relevant approach” with a “clearly defined problem formulation” and for raising their score to 4 (accept).

We would like to emphasize that the novelty of our paper lies in its formal retrieval semantics for natural-language intent, in which a typed first-order STL program both selects and ranks multi-agent scenes. Its relevance to the ICLR community lies in providing a verifiable, label-free neuro-symbolic approach to scenario mining for autonomous driving. We respectfully ask the Area Chair to consider these contributions and the additional evidence when making the final decision.

As a quick summary, we have made the following efforts to respond to and clarify the concerns raised by each reviewer:

  • B26P: The reviewer has acknowledged that our new experiments have "substantially addressed concerns about the baselines and clarified the role of STL." We would like to emphasize the novelty and significance of our work. Specifically, to our knowledge, our work is the first to define the retrieval and ranking of multi-agent scenes from natural-language intent using the robustness of a typed first-order STL program, maximized over agent bindings and time, toward reliable, label-free scenario mining for autonomous driving. Unlike previous work, which generates free-form code without temporal operators or scores (RefAV), translates language into formulas over abstract propositions without grounding them in trajectories (NL2TL), or learns queries from labeled examples (Quivr), we uniquely bind typed agent variables across signal, relational, and map predicates, so that every query has well pre-defined semantics, is validated before execution, and ranks scenes without labels or an LLM at query time. Furthermore, Intent2Query significantly outperforms all 12 baselines in F1 and precision on LADDMS, with its largest margins on medium and hard intents. In addition, reviewers 7VXw and 9KHZ both noted the value of such formalized programs for data mining and safety verification in autonomous driving. Thus, we believe our proposed approach is novel and significant.
  • 9KHZ: We have provided detailed responses to each concern raised in the review (novelty relative to RefAV and NL2TL, benchmark scale, INTERACTION performance and runtime, parsing scale, and the oracle result), including a comparison with RefAV as a new baseline, which Intent2Query significantly outperforms on all three datasets.
  • 7VXw: The reviewer has found our rebuttal helpful in providing additional experimental comparisons with a larger-scale LLM judge and stronger baselines.

Additionally, we have added the following new content, evaluations, and discussions:

  • Stronger baselines: A 14B LLM judge, RefAV program synthesis using its own pipeline, a fine-tuned embedder, four additional zero-shot embedders, and a stronger reranker.
  • Expanded evaluation: Parsing evaluated on all 300 intents and ranking evaluated at different retrieval depths.
  • Label and annotation descriptions: LADDMS labels were initialized using an executable reference labeler and independently verified by two domain experts, with disagreements resolved by a third expert. NGSIM and INTERACTION labels were generated using the same reference labeler.
  • Reproducibility: An anonymized package containing the code, benchmark, and per-intent results that can regenerate every table.

In summary, our contributions include:

  • Clear evidence against stronger baselines. The 12 baselines span example-based retrieval, embeddings, LLM judges, temporal-logic translation, and LLM program synthesis. Intent2Query significantly outperforms all 12 in both F1 and precision on LADDMS, and in 70 of the 72 (12 baselines × 3 datasets × 2 metrics: precision and F1) comparisons across three datasets; the two exceptions are ties with the 14B judge. Intent2Query uses no labels, fine-tuning, or LLM inference at query time.
  • The gains are greatest where the problem is hard. On LADDMS medium and hard intents, ΔF1 is +0.206 over RefAV and +0.168 over the 14B judge.
  • Insights beyond the system. For Intent2Query, precision remains at 0.89–0.92 from easy to hard intents, while the strongest LLM-based baselines lose about half of their precision. Typed agent variables matter more than any other component (−0.325 F1 without them). Larger LLMs and additional training labels cannot replace a well-designed representation, as methods trained on labeled data often struggle to generalize across datasets.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.