- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Intent2Query, which translates natural-language descriptions of traffic interactions into a typed query language grounded in signal temporal logic (STL), then executes those queries over multi-agent trajectories. It reports improved retrieval performance across three datasets.
Critical strengths
The problem is well motivated. Finding specific interactions in large trajectory datasets is useful for autonomous-driving research, and a natural-language interface could reduce the effort of writing search scripts. Typed participants and map-aware predicates are sensible design choices, with substantial gains reported on LADDMS.
Critical weaknesses
The retrieval ground truth needs clearer documentation. Appendix F describes human relevance judgments, but does not specify annotation coverage, how partial and uncertain labels enter the retrieval metrics, or how disagreements were handled. The source of binary relevance labels for NGSIM and INTERACTION is also unclear. These details are essential for interpreting the reported precision and recall.
The baselines are insufficiently specified. Section 5.3 describes the method families, but omits important details such as the embedding and reranking models, scene representations, example selection, and tuning procedures. Without these details, it is difficult to assess baseline strength or reproduce the comparisons.
The technical contribution needs sharper positioning. STL is a reasonable foundation, and the paper acknowledges that quantitative robustness mainly benefits ranking. However, the distinction between the proposed DSL and existing formal trajectory-query approaches remains underdeveloped. The paper should clarify which capabilities are new and which come from integrating established techniques. In general, I'm against abusing STL if the core capabilities of STL are not central to the method.
Questions for authors
How were retrieval labels constructed for each dataset, and what fraction of query–scene pairs received independent human annotation?
Which exact implementations and configurations were used for the baselines?
2 comments
Follow up
Dear Reviewer B26P,
We sincerely thank you for your thoughtful review and for recognizing the importance of the problem and the value of typed, map-aware querying. Your feedback has helped us improve the manuscript, particularly by clarifying the annotation procedure, providing detailed baseline configurations, and sharpening the technical novelty and central role of STL.
We kindly invite you to review our detailed rebuttal and revised manuscript before the discussion period closes. We hope these revisions address your concerns about technical contributions, and we would greatly appreciate your reconsideration of the evaluation. Please let us know if further clarification would be helpful.
Best regards,
Authors of Submission 35256
Response to Reviewer B26P
We thank the reviewer for the careful and constructive review.
Summary. We have uploaded a revised manuscript clarifying retrieval labels, documenting all 12 baselines (Appendix H), adding stronger comparisons (a 14B LLM judge, RefAV, alternative text embed models and fine-tuned embedder), and sharpening our technical positioning. Below, we address annotation reliability (W1), baseline reproducibility (W2), and the novelty and central role of STL (W3).
W1, Q1 (Retrieval labels and annotation). LADDMS labels were initialized using an independent executable reference labeler and subsequently verified through human annotation. Two domain experts independently labeled query–scene pairs based on the reference results, with disagreements resolved by a third expert. NGSIM and INTERACTION labels were generated using the same reference labeler.
Only LADDMS supports ranking evaluation with human-annotated graded relevance (exact/partial/negative/unsure). For nDCG@k, exact/partial/negative receive gains of 3/1/0, while unsure pairs are excluded; precision, recall, and F1 use binary relevance.
W2, Q2 (Baseline implementations and configurations). Appendix H now documents all 12 baselines, including input representations (z-scored scene features, trajectory signals, and scene captions), model identities and prompts (Qwen2.5-family LLMs and open-source Hugging Face embedders; Table 16), and evaluation protocols. To ensure fair comparisons:
Under this protocol, Intent2Query significantly outperforms all baselines on LADDMS, including:
W3 (Technical novelty and the role of STL). We clarify three contributions: (1) typed multi-agent grounding, with distinct, type-constrained agent variables shared across signal, relational, and map predicates; (2) formal retrieval semantics, maximizing scene-level robustness over valid bindings and observation times; and (3) validated natural-language querying, with deterministic program validation.
We do not claim novelty in bounded discrete-time STL robustness or semantic parsing to temporal logic; rather, we integrate them into a unified retrieval framework. Unlike Quivr, which synthesizes queries from labeled examples, Intent2Query requires none: our QBE-STL baseline achieves F1 = 0.453 with 20 labeled scenes per intent, versus 0.738 for Intent2Query. Scenic-based approaches require expert-written programs, while STREL and Traffic Scenario Logic primarily target monitoring or scenario generation.
STL is central, not auxiliary: bounded temporal operators support ordering and metric-time constraints required by 91/300 intents, while quantitative robustness enables principled ranking and adjustable retrieval thresholds. Robustness ranking improves nDCG@10 by +0.050 (95% CI [+0.028, +0.073]) over Boolean matching, and removing temporal operators reduces F1 by 0.057 (Table 12). Adjusting the robustness threshold also enables precision–recall trade-offs without retraining.
In summary, STL provides the established temporal and quantitative foundation; our contribution lies in extending it with typed multi-agent grounding, formally defined retrieval semantics, and validated natural-language querying.
We thank the reviewer again for the valuable feedback, which has strengthened the manuscript. We hope these revisions address the concerns and would appreciate reconsideration of the rating.