- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a dataset and a post-training method to improve LLMs’ capabilities on time series data science analysis problems.
Critical strengths
- The dataset can serve as a good resource for the broader data science agent community.
- The evaluated candidates include frontier models.
Critical weaknesses
Dataset taxonomy: Although Tables 16, 18, and 19 provide selected breakdowns by workflow structure, tool category, computed statistics and data domain, the authors should attempt to provide a clear taxonomy of the generated dataset based on the specific time series analysis skills being evaluated. Without this, it is hard to judge whether this dataset resembles practical workflows. Beyond the existing breakdowns for RITS, please provide a deeper analysis of what can already be achieved by frontier models and whether the post-trained model improves on their performance within those categories.
Cost: Can you provide the cost in terms of tokens and dollars for closed-source LLMs? Specifically, it would be meaningful to know the cost of achieving the reported performance using closed-source models. Comparing these costs would help justify the complexity of training a model.
In Table 4, the closed-source baselines are GPT-4.1 and GPT-4o. Can we evaluate the frontier models used in Table 1 under a comparable tool-access and inference-budget setting so that we understand the gap between RITS and frontier models on these external benchmarks?
It remains unclear how structurally different the OOD templates are from those used in training. Could the authors characterize this overlap such as by reporting the proportion of held-out DAGs whose topology appears in training when tool names and arguments are removed, and whose tool-labeled structure appears when only instance-specific arguments are removed?
Questions for authors
See Weaknesses
5 comments
Response to Reviewer MpY2 (2/3)
2. Performance, tokens, and dollar cost
We estimated token usage and API cost for the closed-source evaluations reported in Table 1, combining ID and OOD questions.
| Model | ID accuracy (%) | OOD accuracy (%) | Mean input tokens/question | Mean output tokens/question | USD/1,000 questions |
|---|---|---|---|---|---|
| RITS (4B) | 83.1 | 69.5 | ≈18,671 | ≈870 | — |
| GPT-5.6-sol | 52.2 | 67.4 | 11,023 | 425 | 39.47 |
| Gemini 3.1 Pro | 53.8 | 67.8 | 19,464 | 1,162 | 52.87 |
| Claude Opus 4.8 | 64.4 | 84.6 | 27,281 | 1,155 | 113.37 |
GPT uses the fewest input and output tokens and has the lowest estimated closed-model cost. Gemini achieves similar accuracy at a higher cost. Claude achieves the highest closed-model accuracy on both splits and has the highest dollar cost, about 2.9 times GPT’s cost, as well as the most input tokens.
Our post-training produces a 4B model with higher ID accuracy than all three closed models and OOD accuracy close to GPT and Gemini. RITS uses fewer input and output tokens per question than Gemini and Claude.
For each question, cost in USD is Σ(tokens × unit rate) / 1,000,000, summed across model calls. Unit rates are in USD per million tokens. We report the mean cost per 1,000 questions.
In the revised manuscript, Appendix G.2, Table 18, reports the token and dollar-cost comparison. The same subsection provides the cost calculation procedure.
3. Frontier-model performance on external benchmarks
Following your suggestion, we evaluated the three frontier models from Table 1 on the external benchmarks, using GPT-5.6-sol, Gemini 3.1 Pro, and Claude Opus 4.8. We evaluated them alongside RITS on the same multiple-choice questions, with access to the same retrieved analysis tools and CSV-series data.
| Task | Questions | RITS accuracy (%) | GPT accuracy (%) | Gemini accuracy (%) | Claude accuracy (%) |
|---|---|---|---|---|---|
| TimeOmni-1 causality | 1,600 | 45.56 | 44.56 | 43.00 | 55.69 |
| TimeOmni-1 scenario | 1,099 | 50.41 | 67.79 | 87.90 | 86.26 |
| TimeOmni-1 decision | 461 | 19.09 | 6.07 | 44.03 | 33.41 |
| MTBench weather | 1,363 | 44.90 | 45.71 | 54.51 | 59.65 |
| MTBench finance | 945 | 78.62 | 82.54 | 89.10 | 92.91 |
RITS achieves accuracy close to GPT on causality and weather questions. The largest gaps occur in scenario understanding, where Gemini and Claude lead RITS by 37.49 and 35.85 percentage points, respectively. Both also achieve higher accuracy on decision, weather, and finance questions. These results show how the gap depends on the analytical task.
In the revised manuscript, Section 4.4 and Table 4 report this frontier-model comparison, using the full TimeOmni-1 ID+OOD scope. Appendix G.6 specifies the evaluated scopes in Table 31 and describes tool access and evaluation settings.
Response to Reviewer MpY2 (3/3)
4. Training–OOD overlap in DAG topology and tool-labeled structure
We measured the structural overlap between training and OOD workflows at the two levels you suggested. Our analysis confirms that the OOD workflows introduce new tool-labeled structures, even when their underlying dependency patterns are shared with training.
We compared the full DAGs, preserving all dependency edges and the data-loading call, at two levels:
- Topology: Remove tool names and arguments, retaining only the dependency structure.
- Tool-labeled structure: Remove only instance-specific arguments, retaining tool names and fixed analytical settings.
Each cell reports the number of distinct OOD DAGs matching training, out of all distinct OOD DAGs at that comparison level. Different tool-labeled structures can share the same topology.
| OOD subset | Topology overlap | Tool-labeled structure overlap |
|---|---|---|
| Unseen-template | 6/7 (85.7%) | 0/15 (0%) |
| Unseen-tool | 6/6 (100%) | 0/16 (0%) |
None of the complete OOD tool-labeled structures appears in training. A familiar dependency pattern alone does not determine which tools and analytical settings to use, which data each call should analyze, or how earlier results should guide later calculations.
To examine whether RITS can handle unseen workflow structures, we further evaluated answer accuracy across two OOD subsets.
| OOD subset | Base accuracy (%) | RITS accuracy (%) |
|---|---|---|
| Seen topology, unseen tool-labeled structure | 34.5 | 68.5 |
| Unseen topology | 33.3 | 86.4 |
RITS results are averaged over three training runs. The unseen-topology subset contains 27 questions.
RITS achieves substantially higher accuracy than Base on both subsets. These results support RITS's ability to transfer learned analytical procedures to unseen tool-labeled structures, with additional evidence of transfer to new dependency patterns.
In the revised manuscript, Appendix C.5 defines the overlap measures. Appendix G.4, Table 26, reports the structural overlap results, with subgroup performance in the accompanying text.
References
[1] Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts, 2021.
Response to Reviewer MpY2 (1/3)
Thank you for recognizing the dataset's potential value to the broader data science agent community and for highlighting the inclusion of frontier models in our evaluation. We appreciate your suggestions and have conducted additional analyses to address your four concerns. Below, we clarify the analytical skills covered by RITS and their practical relevance, compare model performance and cost, extend the external benchmark evaluation to frontier models, and examine the structural overlap between training and OOD workflows.
1. Skill taxonomy, practical workflows, and model performance by category
We organize RITS questions into eight analytical skills by analytical objective. These cover common time-series tasks [1].
RITS also evaluates how these skills combine in multi-step workflows. In practical analysis, an earlier result often determines what to analyze next. In Figure 1, the agent compares two signals’ variability, then computes the selected signal’s mean in another time window. The first result determines which data the next calculation uses. RITS evaluates whether an agent follows these workflows with the correct signals, windows, and intermediate results.
Questions can involve several skills, so category counts overlap. We assign skills from the quantities and analytical settings in reference tool calls. The last two categories are evaluated only on ID.
On OOD questions, RITS improves over Base and SFT in all six represented skills and exceeds GPT and Gemini in five. The table reports answer accuracy (%) on the pooled unseen template and unseen tool splits. N counts questions per category. Base is Qwen3-4B, also used by SFT and RITS. RITS results average three training runs. GPT, Gemini, and Claude denote GPT-5.6-sol, Gemini 3.1 Pro, and Claude Opus 4.8.
Statistical testing is a relative strength, with RITS reaching 83.33%, above the frontier models’ 79.63–80.56%. RITS also improves over SFT on complexity features, from 68.42% to 83.77%, for workflows involving persistence and irregularity.
Claude leads in trend and seasonality, distribution and variability, and complexity features. Cross-series relationships show the largest OOD gap, with 30.41% for RITS versus 100% for Claude. GPT and Gemini also outperform RITS in this category. The anomaly/change comparison covers 14 OOD questions from one template.
On familiar ID workflows, RITS exceeds all three frontier models in five of the eight skill categories.
For cross-series relationships, RITS achieves 96.41% on ID, close to the frontier models’ 95.38–98.46%. Its lower OOD performance identifies difficulty transferring this skill to new analytical workflows.
In the revised manuscript, Appendix C.4, Table 7, presents the skill taxonomy, analytical objectives and ID/OOD question counts. Appendix C.3 explains the workflow design, and Appendix G.5, Table 29, reports the OOD skill-level comparisons. The full ID skill-level comparison is provided in this response.