acceptodds
Under review as a conference paper at ICLR 2027

TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases

11 views

Abstract

Large language models (LLMs) have significantly advanced natural language querying over relational databases, yet their ability to query time-series databases (TSDBs) remains largely unassessed. Existing benchmarks fail to adequately capture the non-unified query syntaxes, diverse application domains, and unique time-specific query intents inherent to TSDBs. To address this gap, we introduce TQTS-BENCH, a multi-syntax benchmark for evaluating text-to-query capabilities over TSDBs. TQTS-BENCH contains 6,125 high-quality question-answering (QA) pairs spanning 97 TSDBs, 23 distinct query syntaxes, 22 application domains, and 4 types of time-specific query intents. It is constructed through a human-centric AI-assisted workflow, where all QA pairs are carefully reviewed and revised by domain experts to ensure quality and correctness. Extensive evaluations of advanced LLMs and state-of-the-art text-to-query methods reveal challenges in querying TSDBs. Even the best-performing model evaluated, Claude-Opus-5, achieves only 48.98% execution accuracy, while humans reach 87.34%. Error analysis reveals that this performance gap mainly stems from the heterogeneous query syntaxes across different TSDBs, misinterpretation of time-specific intents, and incorrect schema linking. These findings highlight new opportunities to narrow the gap between current LLM capabilities and the requirements of TSDB queries in real-world applications. The benchmark is available at: https://anonymous.4open.science/r/TQTS-Bench-00CD.

Primary area:
datasets and benchmarks
Keywords:
time-series database, benchmark, text-to-query, multi-syntax

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NNFYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TQTS-Bench for text-to-query over time series databases. It covers multiple TSDB systems, query syntaxes, application domains, and temporal query intents. And the paper also introduces manual verification. The main finding is that current LLMs still perform far below humans, with syntax heterogeneity, temporal intent understanding, and schema linking identified as major failure sources.

Critical strengths

S1. The problem is well motivated. The fragmented query languages and temporal operations are common.

S2. Human verification is important compared with purely synthetic benchmark generation.

S3. The benchmark has broad coverage across systems, syntaxes, domains, and temporal intents. And the paper provides useful diagnostic analysis.

Critical weaknesses

W1. The introduction overstates the distinction between RDB and TSDB querying. Temporal operations are not unique to TSDBs.

W2. Most questions are generated with LLM assistance and then verified by humans. This is important and ensures correctness. But it is less clear how closely the language distribution reflects real user queries.

W3. The difficulty definition based only on the number of intents is coarse. The authors should also consider syntax rarity, schema size, nesting, or operator complexity.

W4. The paper attributes failures to domain shift, temporal intents, or syntax, but these factors are often confounded. The SQL conversion ablation shows syntax is important, but does not establish that question difficulty is unimportant.

W5. The error analysis is conducted only on a relatively weak model. What about results on the strongest models?

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer NNFY (Part 1 / 2)

Dear Reviewer NNFY,

We thank you for your time, effort, and insightful comments. Please find our responses below.

W1: The introduction overstates the distinction between RDB and TSDB querying. Temporal operations are not unique to TSDBs.

Thank you for pointing this out. We agree that RDBs also support temporal queries and the introduction overstated the distinction between RDBs and TSDBs. We intended to emphasize the role of time in each system. In RDBs, time is typically stored as column values, and SQL supports operations on those values. However, TSDBs organize and store data around time and often provide dedicated functions or syntax for common time-series operations, such as time bucketing and resampling. We will revise the introduction to clarify this point.

W2: It is less clear how closely the language distribution reflects real user queries.

Thank you for raising this point. To characterize the language distribution of TQTS-Bench relative to real-world user questions, we collected 1,000 real user questions from official TSDB forums and blogs and compared lexical and semantic diversity across question sources, including existing TSDB benchmarks TEFD and PromCopilot. We measured lexical diversity using HD-D [1], which is based on the probability that each word type appears at least once in a fixed-size random sample of tokens. Following ChartQAPro [2], we measured semantic diversity as the mean pairwise cosine distance between question embeddings. As shown in the table below, TQTS-Bench exhibits comparable lexical 0\.8631 vs\. 0\.8688 and semantic diversity 0\.2355 vs\. 0\.2427 to real-world user questions, with both scores higher than those of TEFD and PromCopilot. We will include these results and revise the manuscript to make this point clear.

Question sourceNumber of questionsLexical diversity ↑\uparrowSemantic diversity ↑\uparrow
Real-world user questions1,0000.86880.2427
TEFD8,4470.73110.1800
PromCopilot2800.75780.1971
TQTS-Bench6,1250.86310.2355

[1] Expectation identity of the hypergeometric distribution and its application in the calculations of high-order origin moments. Communications in Statistics-Theory and Methods, 2023.

[2] ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering. Findings of ACL 2025.

W3: The difficulty definition based only on the number of intents is coarse.

Thank you for raising this point. To support a more detailed analysis of query difficulty, we added four additional attributes, including syntax rarity, schema size, nesting depth, and operator complexity. For each attribute, we defined criteria for assigning queries to different difficulty levels based on their attribute values.

The table below describes how we compute each attribute and summarizes its distribution. We will also provide a metadata.json file for each QA pair containing these attributes, so researchers can analyze model performance along each dimension.

AttributeCalculation methodRangeEasyMediumHard
Intent countNumber of query intents in the question.1–91–3 (31.72%)4–5 (44.33%)6–9 (23.95%)
Syntax rarityDB-Engines popularity score of the corresponding TSDB management system [1], used as a proxy for syntax rarity. Lower scores indicate greater inferred rarity.0.00–20.4510.39–20.45 (36.72%)1.41–10.38 (34.71%)0–1.40 (28.57%)
Schema sizeTotal number of tokens in the database schema.360–850,028360–1,638 (31.10%)1,639–6,865 (38.94%)6,866–850,028 (29.96%)
Nesting scoreGold-query nesting depth, min–max normalized within each query syntax.0–1.00–0.10 (36.77%)0.11–0.32 (33.92%)0.33–1.0 (29.31%)
Operator complexityNumber of operations in the gold query, such as aggregations, window operations, and sorting.1–4001–8 (33.06%)9–16 (36.95%)17–400 (29.99%)

[1] https://db-engines.com/en/ranking/time+series+dbms

AuthorsOpenReview ↗

Response to Reviewer NNFY (Part 2 / 2)

W4: The paper attributes failures to domain shift, temporal intents, or syntax, but these factors are often confounded. The SQL conversion ablation shows syntax is important, but does not establish that question difficulty is unimportant.

Thank you for pointing this out. We agree that the SQL conversion ablation study does not establish that question difficulty is unimportant. To further investigate whether question difficulty influences the gains from SQL conversion, we compared execution accuracy before and after conversion across three difficulty levels, using the same questions as in the original ablation study.

As shown in the table below, SQL conversion improves accuracy across all groups Easy:0Easy: 0% → 36\.36%; Medium: 0% → 25\.35%; Hard: 0% → 40\.00%, providing further evidence of the importance of syntax. Meanwhile, the varying gains suggest that other factors may also influence performance. We acknowledge that these differences may be confounded by database domains and query intents, and will revise the manuscript to avoid attributing performance differences to any single factor.

DifficultyOriginal %After conversion %
Easy0.0036.36
Medium0.0025.35
Hard0.0040.00
Overall0.0032.21

W5: The error analysis is conducted only on a relatively weak model. What about results on the strongest models?

Thank you for pointing this out. We extended our error analysis to Claude-Opus-5, GPT-6-Sol, and Gemini-3.7-Flash, the three strongest models in our evaluation. We used the same error categories as in the original analysis and calculated the percentage of incorrect cases in each category for each model. As shown in the table below, query syntax errors were the most frequent category for all three models, followed by intent understanding errors. This ordering matches the original analysis of DeepSeek-V4-Pro, indicating that the two main challenges persist even among the strongest models we evaluated.

ModelQuery syntax errors %Intent understanding errors %Schema linking errors %
DeepSeek-V4-Pro75.2961.4317.09
Claude-Opus-558.7655.4713.50
GPT-6-Sol68.1247.8316.30
Gemini-3.7-Flash62.7653.1012.41

Finally, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. If you have any questions, please don't hesitate to reach out to us.

Thanks for the response. W2: I'm still not fully convinced. Similar diversity scores don't really mean the questions are close to real user questions. They can still be quite different in how people actually ask things. W3: the new attributes are useful. But I would also like to see how they relate to model performance. W4: I still have the same concern. The new experiment uses the original failed cases, so the 0% is expected. I agree syntax matters, but it's hard to tell how much the improvement comes from syntax solely.

Reviewer_VBgcOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

One-line Summary.

This paper introduces TQTS-Bench, a text-to-query benchmark over time-series databases (TSDBs). It requires LLMs to support query writing with multiple syntaxes, as different TSDB systems require different syntaxes.

 

Limitation of Prior Work.

  • RDB benchmarks (e.g., Spider, BIRD): grounded in a single SQL syntax and ignoring time-specific query requirements.
  • TSDB benchmarks (e.g., TEFD, PromCopilot): limited to a specific TSDB and a single query syntax.

 

Core Idea.

To build a text-to-query benchmark over TSDBs with multiple syntaxes, two steps are used:

  • TSDB construction: 23 TSDB management systems are selected from the DB-Engines ranking.
  • QA construction and verification: nine query intents are included, and human annotators verify the constructed QA.

 

Effect.

On TQTS-Bench, the best-performing model, Claude-Opus-5, reaches only 48% execution accuracy against 87% for humans, while RDB methods perform far worse.

Critical strengths

Strengths.

(1) Clear positioning and timely problem formulation

Text-to-query over TSDBs is an important yet comparatively underexplored problem. The proposed benchmark, positioned as the first to cover multiple query syntaxes, is novel.

 

(2) Well-designed construction pipeline with human cross-verification

The construction pipeline is well designed. I particularly like that correctness is verified entirely by humans rather than by an LLM-as-judge, which many recent benchmarks rely on by default.

 

(3) Solid experiments with recent models and a fine-grained error analysis

The evaluation covers recent open- and closed-source models as well as representative RDB and TSDB methods (Table 1). The error analysis in Sec. 4.3 is broad and helpful.

Critical weaknesses

Weaknesses.

(1) Lack of per-syntax evaluation for the multi-syntax claim

The benchmark's core benefit is its coverage of 23 query syntaxes, yet no experimental result is broken down by syntax (i.e., to answer the question "what is the most challenging syntax?"). I think including this evaluation would strengthen the central claim on why covering different syntaxes is important.

 

(2) Insufficient reporting of the human verification statistics

In my view, the human cross-verification is one of the main strengths of this benchmark. However, the paper reports only aggregate effort (e.g., over 600 hours), which is insufficient to assess how meaningful the verification was.

I encourage the authors to clarify this setup further, for example by including the agreement rate between the two annotators and the number of QA pairs ultimately discarded, which are commonly reported in other benchmarks that use human verification.

 

(3) The benchmark shows a good diagnosis, but does not hint at the way to potential performance gains

While this is not a major weakness, I personally think it is important for a benchmark to guide how to improve TQTS performance beyond only diagnosing failures. Can the authors briefly demonstrate how the benchmark can be utilized to improve performance (e.g., fine-tuning or few-shot retrieval)? Or at least, discussion on this point would be very helpful to readers.

 

Overall, the work is solid with good positioning and novelty. My current rating is driven mainly by (1), and I would be glad to raise the score if these points are sufficiently addressed in the rebuttal.

Questions for authors

Q. Framing of domain coverage in Figure 1(b)

As a peer working on both RDBs and TSDBs, I do not think the "narrow vs. broad" domain contrast in Fig. 1(b) is the right framing.

In my opinion, RDBs are the more general and broadly used concept. What differs is that the two are built for different purposes, general-purpose vs. time-intensive domains. Because TSDBs store time information by design, they suit time-axis analyses, while RDBs serve as a more general system since they do not strictly require time to be stored. How about reframing Fig. 1(b) as "general domains vs. time-specific/historical domains" instead of "narrow vs. broad"?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VBgc (Part 1 / 2)

Dear Reviewer VBgc,

We thank you for your time, effort, and insightful comments. Please find our responses below.

Q1: Framing of domain coverage in Figure 1bb.

Thank you for raising this point. We agree that the “narrow vs. broad” framing in Fig. 1bb does not accurately reflect the distinction between RDBs and TSDBs. We will replace them with "general-purpose" and "time-intensive," respectively, and adjust the figure and paper accordingly to better reflect their application focuses and differences.

W1: Lack of per-syntax evaluation for the multi-syntax claim.

Thanks for pointing this out. We have reported the execution accuracy of the 23 syntaxes across the three strongest models in the table below. Because some syntaxes have no standalone name, we label each query syntax by its corresponding TSDB management system.

As shown in the table below, the models achieved the highest mean accuracy on CnosDB 794˙679\.46%, GreptimeDB 643˙364\.33%, and InfluxDB 3 Core 623˙262\.32%, and the lowest on M3DB 304˙630\.46%, Machbase Neo 299˙029\.90%, and OpenMLDB 154˙415\.44%. OpenMLDB was the most challenging one for the three evaluated models. This spread illustrates the value of the multi-syntax design: performance on a few syntaxes provides only a partial picture of how well models generate queries across TSDB management systems.

TSDB management systemClaude-Opus-5GPT-6-SolGemini-3.7-FlashMean (descending order)
CnosDB80.54%82.16%75.68%79.46%
GreptimeDB75.44%52.63%64.91%64.33%
InfluxDB 3 Core64.55%62.21%60.20%62.32%
CrateDB59.76%58.54%60.16%59.49%
QuestDB66.23%57.05%53.11%58.80%
IBM Db2 Event Store63.41%45.93%63.01%57.45%
Arc65.52%55.17%50.00%56.90%
TimescaleDB57.83%53.24%52.19%54.42%
BangDB58.62%48.28%53.45%53.45%
Apache Druid48.13%43.57%58.51%50.07%
Riak TS45.24%69.05%30.95%48.41%
InfluxDB OSS v251.17%46.28%47.56%48.34%
ArcadeDB53.97%38.10%52.38%48.15%
VictoriaMetrics44.38%49.44%43.82%45.88%
Prometheus42.46%49.60%38.39%43.49%
openGemini40.00%43.08%33.85%38.97%
DolphinDB48.69%30.07%34.64%37.80%
Apache IoTDB36.51%33.73%29.76%33.33%
TDengine15.26%42.97%38.55%32.26%
GridDB34.31%35.56%24.27%31.38%
M3DB41.38%12.07%37.93%30.46%
Machbase Neo38.54%25.91%25.25%29.90%
OpenMLDB20.00%16.32%10.00%15.44%
AuthorsOpenReview ↗

Response to Reviewer VBgc (Part 2 / 2)

W2: Insufficient reporting of the human verification statistics.

Thank you for this suggestion. We agree that total annotation time alone does not show how human verification shaped the final dataset. Of the 6,133 candidate QA pairs independently reviewed by both annotators, they agreed on 2,466 402˙140\.21%. Among the remaining pairs, the adjudicator accepted 1,056 172˙217\.22% as written, revised and retained 2,603 424˙442\.44%, and discarded 8 01˙30\.13% as unanswerable. All percentages are calculated over the 6,133 candidate pairs. The final dataset contains 6,125 QA pairs. We will report these statistics in the revised manuscript.

W3: The benchmark shows a good diagnosis, but does not hint at the way to potential performance gains.

Thank you for this suggestion. We propose two ways to improve performance and conducted two experiments on all 600 cases used in Sec. 4.3 to briefly validate their effectiveness.

11 Providing syntax and function information. We first collected the syntax and function documents of all the 23 syntaxes from their official websites. Then, we split the documents into chunks of at most 750 tokens, embedded the chunks using a widely-adopted embedding model BGE-M3 [1]. For each question, we used cosine similarity to retrieve the top 4 most relevant chunks and added them to the model’s input. As shown in the table below, providing the syntax and function documentation increased execution accuracy from 27.83% to 32.83% +50˙0percentagepoints+5\.00 percentage points.

SettingEX %
Original27.83
w/ syntax and function information32.83 +50˙0+5\.00

22 Refining queries with execution-guided refinement. For each query that failed to execute, we executed the original query and fed the database error message eg˙,˙acompilationerrore\.g\., a compilation error back to the model in a follow-up turn. We then executed the revised query. If execution failed again, we sent the new error message to the model and asked it to revise the query again. We repeated this process until the query executed successfully or the model had made three revisions. As shown in the table below, execution-guided refinement increased execution accuracy from 27.83% to 34.00% +61˙7percentagepoints+6\.17 percentage points.

SettingEX %
Original27.83
w/ execution-guided refinement34.00 +61˙7+6\.17

[1] M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint, 2024.


Finally, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. If you have any questions, please don't hesitate to reach out to us.

Thank you for the detailed response. The authors have fully addressed my concerns, and I will raise my score to 4 accordingly.

One note: since ICLR allows paper revisions during the discussion period, I had hoped to see the revised framework figure (Fig. 1(b)) in the updated manuscript. I trust that the authors will incorporate this change in the final version.

Dear Reviewer VBgc,

Thank you for your positive feedback, raising your score and paper revision notice! We apologize for not including the revised Fig. 1bb in the updated manuscript during the previous response. We have incorporated the revised figure. Thank you again for your constructive suggestions, which have helped us improve the clarity and accuracy of the paper.

Reviewer_WZtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TQTS-Bench, a multi-syntax benchmark for evaluating text-to-query capabilities over time-series databases (TSDBs). Compared with existing benchmarks for relational databases, such as Spider and BIRD, benchmarking text-to-query over TSDBs introduces additional challenges, including diverse query syntaxes, application domains with continuously collected high-frequency data, and time-specific query intents. To address these challenges, TQTS-Bench contains 6,125 text-to-query pairs spanning 23 query syntaxes, 22 application domains, and four time-specific query intents. The benchmark is constructed through a human-centric, AI-assisted workflow to ensure data quality and correctness. The paper conducts extensive experiments on TQTS-Bench, showing the limitations of existing text-to-query methods on TSDBs and providing a detailed error analysis.

Critical strengths

S1. The paper addresses an important problem of evaluating text-to-query capabilities across diverse time-series databases and query languages.

S2. The paper presents a substantial benchmarking effort, with broad coverage of query syntaxes and application domains, extensive experiments, and detailed error analysis.

Critical weaknesses

W1. The novelty of TQTS-Bench over existing TSDB benchmarks is unclear. Prior work has already developed benchmarks for Text-to-Flux (TEFD), Text-to-Flux/InfluxQL (Dranca et al.), and Text-to-PromQL (PromCopilot). While TQTS-Bench covers more query syntaxes and application domains, it is unclear what new evaluation capabilities it provides beyond combining these existing settings. The paper should better clarify the benefits of its multi-syntax design.

W2. The baseline comparison is not comprehensive enough. For TSDB-specific methods, the paper only compares with PromCopilot, leaving out relevant approaches such as TEFD and Dranca et al., even on their supported query syntaxes. For RDB-based methods, the comparison includes earlier methods such as DAIL-SQL and DIN-SQL but misses some recent approaches. More representative baselines would make the evaluation more convincing.

W3. The error analysis raises concerns about the evaluation setup. Query syntax errors account for 75.29% of incorrect cases, suggesting that the models may not be sufficiently adapted to TSDB query languages. It is unclear whether the methods are provided with sufficient syntax and function information or use syntax validation and execution-guided refinement. The paper should clarify these settings and examine how many errors can be corrected with such mechanisms.

Questions for authors

Q1. What new evaluation capabilities does TQTS-Bench provide beyond combining existing TSDB benchmarks? Why are relevant methods such as TEFD and Dranca et al. not included in the comparison?

Q2. Given that 75.29% of incorrect cases are query syntax errors, what syntax information and execution feedback are provided to the models? How many of these errors could be corrected through syntax validation or execution-guided refinement?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WZtM (Part 1 / 2)

Dear Reviewer WZtM,

We thank you for your time, effort, and insightful comments. Please find our responses below.

Q1/W1: What new evaluation capabilities does TQTS-Bench provide beyond combining existing TSDB benchmarks?

The main evaluation capability our benchmark adds to existing TSDB benchmarks is the ability to evaluate TQTS under realistic deployment conditions. While SQL dominates relational databases, the time-series database field uses a range of syntaxes, with no single syntax dominating the ecosystem. In practice, the users of an LLM can ask it to generate queries in any of these syntaxes. Therefore, its performance on different syntaxes is expected to be evaluated according to their market shares in realistic deployment conditions. Existing benchmarks Text−to−Flux,Text−to−Flux/InfluxQL,andText−to−PromQLText-to-Flux, Text-to-Flux/InfluxQL, and Text-to-PromQL collectively cover three query syntaxes: Flux, InfluxQL, and PromQL. These syntaxes are used by only 2 (InfluxDB and Prometheus) of the 20 highest-ranked time-series databases in the DB-Engines popularity ranking [1]. Evaluation on these benchmarks alone can therefore substantially overestimate performance across a broader range of syntaxes. For example, GPT-6-Sol achieves 52.70% accuracy on Flux, InfluxQL, and PromQL, but only 44.64% on the 20 additional syntaxes covered by our benchmark, a gap of 8.06 percentage points. Therefore, our benchmark offers a more accurate evaluation of TQTS under realistic deployment conditions.

Flux+InfluxQL+PromQL20 additional syntaxes in our benchmark
GPT-6-Sol execution accuracy52.70%44.64%

[1] https://db-engines.com/en/ranking/time+series+dbms

Q2/W3: Clarify the settings of syntax and function information, as well as syntax validation and execution-guided refinement.

Thank you for pointing this out. In our evaluation, we did not provide syntax and function information to LLMs. We also did not use syntax validation and execution-guided refinement. We only provided the question, database schema, and database context without demonstration examples. To assess the effects of these two settings, we conducted two additional experiments on the cases exhibiting query syntax errors 752˙975\.29% of all incorrect cases.

11 Providing syntax and function information. We collected the syntax and function documents of all 23 syntaxes from their official websites. Given the length of the documentation, we split it into chunks of at most 750 tokens, embedded these chunks using a widely-adopted BGE-M3 embedding model [1], and used cosine similarity to retrieve the top 4 chunks most relevant to each input question. We then added the retrieved chunks to the model’s input. As shown in the table below, 15.64% of the original syntax error cases produced the correct execution result after syntax information was added.

SettingEX %
Original0.00
w/ syntax and function information15.64

22 Using syntax error correction via execution-guided refinement. For each syntax error case, we executed the original query and passed the database execution error message eg˙,˙acompilationerrore\.g\., a compilation error to the model in a follow-up turn. The model revised the query based on this feedback, and we executed the revised query. If execution still failed, we passed the new error message back to the model for another revision, allowing up to three revision turns in total. As shown in the table below, 10.74% of the original syntax error cases produced the correct execution result after refinement.

SettingEX %
Original0.00
w/ execution-guided refinement10.74

[1] M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint, 2024.

AuthorsOpenReview ↗

Response to Reviewer WZtM (Part 2 / 2)

W2: Additional baselines.

Thank you for pointing this out. We clarify our choice of TSDB baselines and report additional results for recent TSDB and RDB methods.

11 TSDB baselines and additional results. TEFD [1] is a text-to-Flux benchmark for InfluxDB. Its FluxEngine pipeline is used to construct the dataset, so we did not include it as a query generation baseline. Dranca et al. [2] fine-tuned separate LoRA adapters for Flux and InfluxQL on a synthetic corpus. We could not locate a public release of the training corpus, training code, or adapter weights, which prevented us from faithfully reproducing their method on TQTS-Bench, so we did not include it as a TSDB method baseline.

To broaden the comparison, we evaluated PromQLAgent May,2026May, 2026 [3], a recent method for generating PromQL queries from natural language. As shown in the table below, PromQLAgent achieved 3.23% overall execution accuracy on TQTS-Bench, further supporting our finding that existing TSDB methods have limited generalizability to TQTS tasks.

TSDB methodEasyMediumHardOverall
PromQLAgent8.08%1.22%0.55%3.23%

22 Additional results for recent RDB methods. We evaluated three recent and representative RDB methods, APEX-SQL (August, 2026) [4], AutoLink (January, 2026) [5], and EvoSQL (October, 2026) [6], under the same experimental settings. As shown in the table below, their overall execution accuracy ranged from 9.24% to 11.05%. These results reinforce our original finding that existing RDB methods struggle with TQTS tasks and transfer poorly to TSDBs.

RDB methodEasyMediumHardOverall
APEX-SQL18.53%6.45%5.04%9.94%
AutoLink19.15%4.94%4.09%9.24%
EvoSQL23.06%6.22%4.09%11.05%

[1] TEFD: A Benchmark for Natural Language to Flux Query Generation in Time-Series Databases. KDD 2026.

[2] LLMs for industrial databases: an agro-food production plant use case. Frontiers in Artificial Intelligence, 2026.

[3] PromQLAgent: A Text-to-PromQL Framework Based on Dual Knowledge Base Retrieval and Adaptive Repair, ICAIBD 2026.

[4] APEX-SQL: Talking to the Data via Agentic Exploration for Text-to-SQL. KDD 2026.

[5] AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale. AAAI 2026.

[6] EvoSQL: Memory-Augmented Critic-Generator Co-Evolution for Text-to-SQL. Findings of EMNLP 2026.


Finally, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. If you have any questions, please don't hesitate to reach out to us.

AuthorsOpenReview ↗

Looking forward to further feedback

Dear Reviewer,

We sincerely thank you again for your recognition of our work. As we noted in our previous response, we have addressed your suggestions in our point-by-point response. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript. If you have any additional comments, we will do our best to address them.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response

Dear Reviewers, ACs and SACs,

Thank you for your insightful and constructive feedback. We are encouraged by the reviewers’ recognition of the importance of evaluating text-to-query capabilities over time-series databases, the broad coverage of TQTS-Bench, its human verification process, and its informative experimental analysis. The strengths highlighted by the reviewers are summarized below.

An important and underexplored evaluation problem: All reviewers recognized the practical motivation for evaluating text-to-query capabilities across heterogeneous time-series databases and query languages.

  • R_WZtM: “The paper addresses an important problem of evaluating text-to-query capabilities across diverse time-series databases and query languages.”

  • R_VBgc: “Text-to-query over TSDBs is an important yet comparatively underexplored problem. The proposed benchmark, positioned as the first to cover multiple query syntaxes, is novel.”

  • R_NNFY: “The problem is well motivated. The fragmented query languages and temporal operations are common.”

Broad coverage across query syntaxes, intents, and application domains: The reviewers acknowledged the substantial benchmarking effort behind TQTS-Bench, which contains 6,125 QA pairs spanning 97 time-series databases, 23 query syntaxes, 22 application domains, and four types of time-specific query intents.

  • R_WZtM: “The paper presents a substantial benchmarking effort, with broad coverage of query syntaxes and application domains, extensive experiments, and detailed error analysis.”

  • R_NNFY: “The benchmark has broad coverage across systems, syntaxes, domains, and temporal intents.”

Human verification for benchmark quality: The reviewers highlighted the value of our human-centered construction and verification process.

  • R_VBgc: “The construction pipeline is well designed. I particularly like that correctness is verified entirely by humans rather than by an LLM-as-judge, which many recent benchmarks rely on by default.”

  • R_NNFY: “Human verification is important compared with purely synthetic benchmark generation.”

Extensive evaluation and informative error analysis: The reviewers appreciated the evaluation of recent models and representative methods, together with the analysis of their limitations.

  • R_VBgc: “Solid experiments with recent models and a fine-grained error analysis.”

  • R_VBgc: “The error analysis in Sec. 4.3 is broad and helpful.”

  • R_NNFY: “The paper provides useful diagnostic analysis.”

We are also grateful for the reviewers’ constructive suggestions. We have provided detailed responses and additional analyses, with the main points summarized below.

Clarifications:

  • Evaluation value of the multi-syntax design beyond existing benchmarks RWZtMR_WZtM.

  • Baseline selection and original evaluation settings RWZtMR_WZtM.

  • Human verification statistics and adjudication outcomes RVBgcR_VBgc.

  • Revised application-domain framing in Figure 1bb RVBgcR_VBgc.

  • Clarified the role of time in RDBs and TSDBs RNNFYR_NNFY.

  • Evidence and limitations regarding resemblance to real-user questions RNNFYR_NNFY.

  • Difficulty characterization beyond intent count RNNFYR_NNFY.

  • Interpretation and limitations of SQL-conversion ablation RNNFYR_NNFY.

Additional Results:

  • Experiments with syntax documentation and execution-guided refinement, including syntax-error recovery analyses RWZtM,RVBgc,RNNFYR_WZtM, R_VBgc, R_NNFY.

  • Additional baselines: PromQLAgent, APEX-SQL, AutoLink, and EvoSQL RWZtMR_WZtM.

  • Per-syntax evaluation across all 23 syntaxes for the three strongest models RVBgcR_VBgc.

  • Diversity comparison with 1,000 real-user questions and blinded human evaluation RNNFYR_NNFY.

  • Performance analysis across intent count, syntax rarity, schema size, nesting depth, and operator complexity RNNFYR_NNFY.

  • Extended error analysis on the three strongest models RNNFYR_NNFY.

Following our responses, R_VBgc explicitly confirmed that his concerns had been fully addressed and raised his rating to 4 (Clear acceptance):

“The authors have fully addressed my concerns, and I will raise my score to 4 accordingly.”

We hope that these clarifications and additional results help resolve the reviewers’ questions and strengthen the case for TQTS-Bench as a useful resource for evaluating and improving text-to-query capabilities over time-series databases. We sincerely thank all reviewers for their time and thoughtful feedback, and the ACs and SACs for their amazing effort in organizing the reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.