- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TQTS-Bench for text-to-query over time series databases. It covers multiple TSDB systems, query syntaxes, application domains, and temporal query intents. And the paper also introduces manual verification. The main finding is that current LLMs still perform far below humans, with syntax heterogeneity, temporal intent understanding, and schema linking identified as major failure sources.
Critical strengths
S1. The problem is well motivated. The fragmented query languages and temporal operations are common.
S2. Human verification is important compared with purely synthetic benchmark generation.
S3. The benchmark has broad coverage across systems, syntaxes, domains, and temporal intents. And the paper provides useful diagnostic analysis.
Critical weaknesses
W1. The introduction overstates the distinction between RDB and TSDB querying. Temporal operations are not unique to TSDBs.
W2. Most questions are generated with LLM assistance and then verified by humans. This is important and ensures correctness. But it is less clear how closely the language distribution reflects real user queries.
W3. The difficulty definition based only on the number of intents is coarse. The authors should also consider syntax rarity, schema size, nesting, or operator complexity.
W4. The paper attributes failures to domain shift, temporal intents, or syntax, but these factors are often confounded. The SQL conversion ablation shows syntax is important, but does not establish that question difficulty is unimportant.
W5. The error analysis is conducted only on a relatively weak model. What about results on the strongest models?
Questions for authors
See weaknesses.
6 comments
Response to Reviewer NNFY (Part 2 / 2)
W4: The paper attributes failures to domain shift, temporal intents, or syntax, but these factors are often confounded. The SQL conversion ablation shows syntax is important, but does not establish that question difficulty is unimportant.
Thank you for pointing this out. We agree that the SQL conversion ablation study does not establish that question difficulty is unimportant. To further investigate whether question difficulty influences the gains from SQL conversion, we compared execution accuracy before and after conversion across three difficulty levels, using the same questions as in the original ablation study.
As shown in the table below, SQL conversion improves accuracy across all groups , providing further evidence of the importance of syntax. Meanwhile, the varying gains suggest that other factors may also influence performance. We acknowledge that these differences may be confounded by database domains and query intents, and will revise the manuscript to avoid attributing performance differences to any single factor.
| Difficulty | Original | After conversion |
|---|---|---|
| Easy | 0.00 | 36.36 |
| Medium | 0.00 | 25.35 |
| Hard | 0.00 | 40.00 |
| Overall | 0.00 | 32.21 |
W5: The error analysis is conducted only on a relatively weak model. What about results on the strongest models?
Thank you for pointing this out. We extended our error analysis to Claude-Opus-5, GPT-6-Sol, and Gemini-3.7-Flash, the three strongest models in our evaluation. We used the same error categories as in the original analysis and calculated the percentage of incorrect cases in each category for each model. As shown in the table below, query syntax errors were the most frequent category for all three models, followed by intent understanding errors. This ordering matches the original analysis of DeepSeek-V4-Pro, indicating that the two main challenges persist even among the strongest models we evaluated.
| Model | Query syntax errors | Intent understanding errors | Schema linking errors |
|---|---|---|---|
| DeepSeek-V4-Pro | 75.29 | 61.43 | 17.09 |
| Claude-Opus-5 | 58.76 | 55.47 | 13.50 |
| GPT-6-Sol | 68.12 | 47.83 | 16.30 |
| Gemini-3.7-Flash | 62.76 | 53.10 | 12.41 |
Finally, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. If you have any questions, please don't hesitate to reach out to us.
Thanks for the response. W2: I'm still not fully convinced. Similar diversity scores don't really mean the questions are close to real user questions. They can still be quite different in how people actually ask things. W3: the new attributes are useful. But I would also like to see how they relate to model performance. W4: I still have the same concern. The new experiment uses the original failed cases, so the 0% is expected. I agree syntax matters, but it's hard to tell how much the improvement comes from syntax solely.
Response to Reviewer NNFY (Part 1 / 2)
Dear Reviewer NNFY,
We thank you for your time, effort, and insightful comments. Please find our responses below.
W1: The introduction overstates the distinction between RDB and TSDB querying. Temporal operations are not unique to TSDBs.
Thank you for pointing this out. We agree that RDBs also support temporal queries and the introduction overstated the distinction between RDBs and TSDBs. We intended to emphasize the role of time in each system. In RDBs, time is typically stored as column values, and SQL supports operations on those values. However, TSDBs organize and store data around time and often provide dedicated functions or syntax for common time-series operations, such as time bucketing and resampling. We will revise the introduction to clarify this point.
W2: It is less clear how closely the language distribution reflects real user queries.
Thank you for raising this point. To characterize the language distribution of TQTS-Bench relative to real-world user questions, we collected 1,000 real user questions from official TSDB forums and blogs and compared lexical and semantic diversity across question sources, including existing TSDB benchmarks TEFD and PromCopilot. We measured lexical diversity using HD-D [1], which is based on the probability that each word type appears at least once in a fixed-size random sample of tokens. Following ChartQAPro [2], we measured semantic diversity as the mean pairwise cosine distance between question embeddings. As shown in the table below, TQTS-Bench exhibits comparable lexical 0\.8631 vs\. 0\.8688 and semantic diversity 0\.2355 vs\. 0\.2427 to real-world user questions, with both scores higher than those of TEFD and PromCopilot. We will include these results and revise the manuscript to make this point clear.
[1] Expectation identity of the hypergeometric distribution and its application in the calculations of high-order origin moments. Communications in Statistics-Theory and Methods, 2023.
[2] ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering. Findings of ACL 2025.
W3: The difficulty definition based only on the number of intents is coarse.
Thank you for raising this point. To support a more detailed analysis of query difficulty, we added four additional attributes, including syntax rarity, schema size, nesting depth, and operator complexity. For each attribute, we defined criteria for assigning queries to different difficulty levels based on their attribute values.
The table below describes how we compute each attribute and summarizes its distribution. We will also provide a metadata.json file for each QA pair containing these attributes, so researchers can analyze model performance along each dimension.
[1] https://db-engines.com/en/ranking/time+series+dbms