acceptodds
Under review as a conference paper at ICLR 2027

RITS: Reasoning with Identifiable Traces for Time-Series Analysis Agents

Abstract

In domains such as finance, healthcare, and industrial monitoring, many decisions rely on time-series analysis. Tool-using agents promise to automate such analyses posed as natural-language questions that require a multi-step workflow of analysis tools. However, training and evaluating agents that execute such analyses faithfully is challenging because natural analysis questions often underspecify the intended procedure. Specifically, the same question can be addressed with different tools and parameter bindings, and these choices can yield different answers. There is then no single correct answer to train on or to grade against, and no unique reference workflow exists to check a produced analysis against. We present RITS, a framework that constructs trace-identifiable multi-hop time-series questions and trains and evaluates agents against their reference workflows. We build every question to admit a unique executable workflow within a modeled tool space, and refer to that workflow as a trace. A produced analysis can thus be scored against a single reference. To this end, RITS instantiates templates structured as directed acyclic graphs (DAGs) over 308 parameterized analysis tools grounded in standard libraries. It executes every candidate trace and retains a question only when its constraints determine a single trace. We then post-train a tool-using agent with a trace-aware reward that scores both the procedure and the answer. Experiments show that trace-aware reinforcement learning reaches near-perfect trace fidelity with half the tool calls of answer-only training, while improving answer accuracy. The trained agent generalizes to held-out workflow structures and tools and transfers zero-shot to external benchmarks. Compared with zero-shot frontier agents, it attains comparable answer accuracy with higher trace fidelity and fewer tool calls.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Agent, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MpY2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a dataset and a post-training method to improve LLMs’ capabilities on time series data science analysis problems.

Critical strengths
  1. The dataset can serve as a good resource for the broader data science agent community.
  2. The evaluated candidates include frontier models.
Critical weaknesses

Dataset taxonomy: Although Tables 16, 18, and 19 provide selected breakdowns by workflow structure, tool category, computed statistics and data domain, the authors should attempt to provide a clear taxonomy of the generated dataset based on the specific time series analysis skills being evaluated. Without this, it is hard to judge whether this dataset resembles practical workflows. Beyond the existing breakdowns for RITS, please provide a deeper analysis of what can already be achieved by frontier models and whether the post-trained model improves on their performance within those categories.

Cost: Can you provide the cost in terms of tokens and dollars for closed-source LLMs? Specifically, it would be meaningful to know the cost of achieving the reported performance using closed-source models. Comparing these costs would help justify the complexity of training a model.

In Table 4, the closed-source baselines are GPT-4.1 and GPT-4o. Can we evaluate the frontier models used in Table 1 under a comparable tool-access and inference-budget setting so that we understand the gap between RITS and frontier models on these external benchmarks?

It remains unclear how structurally different the OOD templates are from those used in training. Could the authors characterize this overlap such as by reporting the proportion of held-out DAGs whose topology appears in training when tool names and arguments are removed, and whose tool-labeled structure appears when only instance-specific arguments are removed?

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer MpY2 (1/3)

Thank you for recognizing the dataset's potential value to the broader data science agent community and for highlighting the inclusion of frontier models in our evaluation. We appreciate your suggestions and have conducted additional analyses to address your four concerns. Below, we clarify the analytical skills covered by RITS and their practical relevance, compare model performance and cost, extend the external benchmark evaluation to frontier models, and examine the structural overlap between training and OOD workflows.

1. Skill taxonomy, practical workflows, and model performance by category

We organize RITS questions into eight analytical skills by analytical objective. These cover common time-series tasks [1].

RITS also evaluates how these skills combine in multi-step workflows. In practical analysis, an earlier result often determines what to analyze next. In Figure 1, the agent compares two signals’ variability, then computes the selected signal’s mean in another time window. The first result determines which data the next calculation uses. RITS evaluates whether an agent follows these workflows with the correct signals, windows, and intermediate results.

Analytical skillPractical objectiveID questionsOOD questions
Trend, seasonality and spectraIdentify trends, cycles, and recurring patterns.102262
Distribution and variabilityCompare signal levels, spread, and distributions.347415
Statistical testingCheck assumptions and test hypotheses, including stationarity.69108
Cross-series relationshipsExamine relationships and timing differences between signals.65114
Anomaly and changeDetect unusual observations and changes in signal behavior.8214
Complexity featuresMeasure persistence, irregularity, and scaling behavior.12276
Clustering and multivariate structureIdentify shared patterns and group related variables.300
Transform-parameter estimationEstimate transformations that stabilize scale and variance.30

Questions can involve several skills, so category counts overlap. We assign skills from the quantities and analytical settings in reference tool calls. The last two categories are evaluated only on ID.

On OOD questions, RITS improves over Base and SFT in all six represented skills and exceeds GPT and Gemini in five. The table reports answer accuracy (%) on the pooled unseen template and unseen tool splits. N counts questions per category. Base is Qwen3-4B, also used by SFT and RITS. RITS results average three training runs. GPT, Gemini, and Claude denote GPT-5.6-sol, Gemini 3.1 Pro, and Claude Opus 4.8.

Analytical skillNBaseSFTRITSGPTGeminiClaude
Trend, seasonality and spectra26229.7746.5652.9347.7148.4777.10
Distribution and variability41530.8462.6566.5964.8265.3081.69
Statistical testing10854.6374.0783.3380.5679.6380.56
Cross-series relationships11427.1926.3230.4138.6041.23100.00
Anomaly and change140.0028.5747.620.000.000.00
Complexity features7630.2668.4283.7776.3277.6398.68

Statistical testing is a relative strength, with RITS reaching 83.33%, above the frontier models’ 79.63–80.56%. RITS also improves over SFT on complexity features, from 68.42% to 83.77%, for workflows involving persistence and irregularity.

Claude leads in trend and seasonality, distribution and variability, and complexity features. Cross-series relationships show the largest OOD gap, with 30.41% for RITS versus 100% for Claude. GPT and Gemini also outperform RITS in this category. The anomaly/change comparison covers 14 OOD questions from one template.

On familiar ID workflows, RITS exceeds all three frontier models in five of the eight skill categories.

Analytical skillNBaseSFTRITSGPTGeminiClaude
Trend, seasonality and spectra10222.5527.4579.7420.5932.3542.16
Distribution and variability34723.3436.0277.8133.7235.1649.57
Statistical testing6937.6855.0788.8947.8360.8782.61
Cross-series relationships6578.4692.3196.4195.3896.9298.46
Anomaly and change8234.1559.7673.1753.6643.9051.22
Complexity features12211.4839.3471.0424.5931.9745.90
Clustering and multivariate structure3073.3390.0098.89100.00100.00100.00
Transform-parameter estimation366.67100.00100.00100.00100.00100.00

For cross-series relationships, RITS achieves 96.41% on ID, close to the frontier models’ 95.38–98.46%. Its lower OOD performance identifies difficulty transferring this skill to new analytical workflows.

In the revised manuscript, Appendix C.4, Table 7, presents the skill taxonomy, analytical objectives and ID/OOD question counts. Appendix C.3 explains the workflow design, and Appendix G.5, Table 29, reports the OOD skill-level comparisons. The full ID skill-level comparison is provided in this response.

AuthorsOpenReview ↗

Response to Reviewer MpY2 (2/3)

2. Performance, tokens, and dollar cost

We estimated token usage and API cost for the closed-source evaluations reported in Table 1, combining ID and OOD questions.

ModelID accuracy (%)OOD accuracy (%)Mean input tokens/questionMean output tokens/questionUSD/1,000 questions
RITS (4B)83.169.5≈18,671≈870—
GPT-5.6-sol52.267.411,02342539.47
Gemini 3.1 Pro53.867.819,4641,16252.87
Claude Opus 4.864.484.627,2811,155113.37

GPT uses the fewest input and output tokens and has the lowest estimated closed-model cost. Gemini achieves similar accuracy at a higher cost. Claude achieves the highest closed-model accuracy on both splits and has the highest dollar cost, about 2.9 times GPT’s cost, as well as the most input tokens.

Our post-training produces a 4B model with higher ID accuracy than all three closed models and OOD accuracy close to GPT and Gemini. RITS uses fewer input and output tokens per question than Gemini and Claude.

For each question, cost in USD is Σ(tokens × unit rate) / 1,000,000, summed across model calls. Unit rates are in USD per million tokens. We report the mean cost per 1,000 questions.

In the revised manuscript, Appendix G.2, Table 18, reports the token and dollar-cost comparison. The same subsection provides the cost calculation procedure.

3. Frontier-model performance on external benchmarks

Following your suggestion, we evaluated the three frontier models from Table 1 on the external benchmarks, using GPT-5.6-sol, Gemini 3.1 Pro, and Claude Opus 4.8. We evaluated them alongside RITS on the same multiple-choice questions, with access to the same retrieved analysis tools and CSV-series data.

TaskQuestionsRITS accuracy (%)GPT accuracy (%)Gemini accuracy (%)Claude accuracy (%)
TimeOmni-1 causality1,60045.5644.5643.0055.69
TimeOmni-1 scenario1,09950.4167.7987.9086.26
TimeOmni-1 decision46119.096.0744.0333.41
MTBench weather1,36344.9045.7154.5159.65
MTBench finance94578.6282.5489.1092.91

RITS achieves accuracy close to GPT on causality and weather questions. The largest gaps occur in scenario understanding, where Gemini and Claude lead RITS by 37.49 and 35.85 percentage points, respectively. Both also achieve higher accuracy on decision, weather, and finance questions. These results show how the gap depends on the analytical task.

In the revised manuscript, Section 4.4 and Table 4 report this frontier-model comparison, using the full TimeOmni-1 ID+OOD scope. Appendix G.6 specifies the evaluated scopes in Table 31 and describes tool access and evaluation settings.

AuthorsOpenReview ↗

Response to Reviewer MpY2 (3/3)

4. Training–OOD overlap in DAG topology and tool-labeled structure

We measured the structural overlap between training and OOD workflows at the two levels you suggested. Our analysis confirms that the OOD workflows introduce new tool-labeled structures, even when their underlying dependency patterns are shared with training.

We compared the full DAGs, preserving all dependency edges and the data-loading call, at two levels:

  • Topology: Remove tool names and arguments, retaining only the dependency structure.
  • Tool-labeled structure: Remove only instance-specific arguments, retaining tool names and fixed analytical settings.

Each cell reports the number of distinct OOD DAGs matching training, out of all distinct OOD DAGs at that comparison level. Different tool-labeled structures can share the same topology.

OOD subsetTopology overlapTool-labeled structure overlap
Unseen-template6/7 (85.7%)0/15 (0%)
Unseen-tool6/6 (100%)0/16 (0%)

None of the complete OOD tool-labeled structures appears in training. A familiar dependency pattern alone does not determine which tools and analytical settings to use, which data each call should analyze, or how earlier results should guide later calculations.

To examine whether RITS can handle unseen workflow structures, we further evaluated answer accuracy across two OOD subsets.

OOD subsetBase accuracy (%)RITS accuracy (%)
Seen topology, unseen tool-labeled structure34.568.5
Unseen topology33.386.4

RITS results are averaged over three training runs. The unseen-topology subset contains 27 questions.

RITS achieves substantially higher accuracy than Base on both subsets. These results support RITS's ability to transfer learned analytical procedures to unseen tool-labeled structures, with additional evidence of transfer to new dependency patterns.

In the revised manuscript, Appendix C.5 defines the overlap measures. Appendix G.4, Table 26, reports the structural overlap results, with subgroup performance in the accompanying text.

References

[1] Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts, 2021.

Reviewer_gtHXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents RITS, a framework for constructing, training on, and evaluating multi-hop time-series analysis questions with executable reference workflows. The central motivation of the paper is that final-answer correctness does not establish whether an agent followed the intended analytical procedure, particularly when different tools or parameter settings can produce plausible or coincidentally identical answers. RITS constructs questions intended to identify a unique workflow within a predefined tool space, using 101 DAG templates and 308 parameterized tools spanning 38 analysis categories. Each retained example includes a reference trace, intermediate results, and a final answer. The framework post-trains a Qwen3-4B agent with group-relative policy optimization using rewards for answer correctness, trace fidelity, intermediate-value recovery, and response formatting. Experiments compare the resulting policy with zero-shot models, supervised fine-tuning, and code execution, and evaluate reward ablations, retrieval, controlled out-of-distribution generalization, and external benchmark transfer. The reported results show improvements in execution discipline and in-distribution accuracy, although generalization and transfer are more mixed.

Critical strengths
  1. The paper identifies an important distinction between obtaining the right answer and executing the right analysis.

  2. The dataset construction provides rich, execution-grounded supervision. RITS goes beyond attaching textual reasoning annotations to questions. It records executable calls, parameter bindings, dependencies, intermediate results, and final answers.

  3. The controlled results demonstrate substantial improvements in tool-use reliability.

Critical weaknesses
  1. Although isolating faithful execution is a legitimate objective, the paper does not establish how representative its template-generated questions are of realistic time-series analytics requests. Some examples involve deeply nested combinations of statistics and selection rules without explaining the analytical objective motivating that composition. For example, the five-hop stock-analysis example in Appendix H uses successive Hilbert-envelope comparisons to select a signal and window for a seasonal-strength comparison, but the practical rationale for this workflow is unclear. Thus, the benchmark might primarily measure the translation of explicit procedural descriptions into API calls, rather than establish the broader usefulness of the resulting time-series analysis agent. Domain-expert validation of the templates and evaluation on independently authored, practically motivated analytical requests would strengthen the contribution.

  2. The dataset generates questions from only 26 real series across nine corpora. While this can provide workflow diversity, it does not necessarily ensure comparable diversity in the underlying temporal patterns. Appendix C reports sampling cadences ranging from five minutes to weekly, but does not systematically document each series’ number of observations, temporal coverage, sampling frequency, dimensionality, or contribution to the question pool. These details are important for assessing whether the underlying data adequately support the benchmark’s breadth.

  3. Algorithm 1 processes entire dependency levels and advances a cursor beyond the last call matched in each level. This does not generally preserve equivalence under valid topological reorderings. Consider a DAG with dependencies r --> a, a --> c, and r --> b, where r is the data loading. Its dependency levels are {r}, {a,b}, and {c}, but both (r,a,b,c) and (r,a,c,b) could be valid execution orders. Assume distinct tool names and successful execution of all calls. Under the binary matching rule used for ordered-valid tool F1, the first ordering matches all four calls. In the second, matching a and b advances the cursor beyond c, making that otherwise valid call ineligible. Thus, equivalent executions could receive different scores. The authors should clarify whether the supported template class excludes such cases.

  4. The necessity of the proposed trace-aware reward is not clearly demonstrated beyond simpler intermediate supervision. In Table 2, R3 achieves 85.8% ID accuracy, compared with 84.8% for the full method, and 83.4% OOD accuracy, compared with 84.4%. It also reaches a 99.0% ID trace score without directly optimizing trace alignment. These results support the value of intermediate supervision, but provide weaker evidence for the distinctive benefit of the proposed DAG-matching reward. Repeated runs for the reward variants, evaluation under retrieved candidate sets, and a simpler execution-success-plus-call-cost reward would help determine which gains specifically require trace alignment.

Questions for authors
  1. Are the templates informed by practitioner workflows or domain-expert input, or designed primarily as controlled tests of procedural instruction following?

  2. What constitutes one of the 26 series for dataset construction? Could the authors provide a per-series breakdown of the number of observations, temporal coverage, sampling frequency, dimensionality, and number of generated questions?

  3. Given R3's comparable accuracy, trace scores, and call counts in Table 2, what evidence establishes a reproducible benefit from adding the trace-alignment reward beyond intermediate-value supervision?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gtHX (1/3)

Thank you for highlighting the importance of checking both the answer and the analysis steps. We also appreciate your recognition of the dataset's recorded tool calls and intermediate results for training, and the substantial improvements in tool-use reliability shown in the controlled experiments. Below, we discuss the task design and data coverage, correct the scoring rule, and examine the added value of trace alignment.

1. Template design and practical relevance

Template design. We designed the templates as controlled tasks for reliable multi-step analysis. We chose and combined operations around specific analytical objectives, with explicit methods and inputs so that the required analysis can be checked.

  • Measuring how often and how long values stay above average. The agent first calculates the signal’s mean and then uses it as the threshold for counting above-average observations or finding the longest consecutive run. This connects the two operations for a concrete purpose, with a threshold adapted to the signal and interval. Similar measurements have been used to predict surges in demand for job skills [1].

  • Checking whether a series is stationary. The agent applies ADF and KPSS to the same signal and interval. The tests provide complementary evidence about stationarity, allowing the agent to compare their conclusions. The official statsmodels documentation demonstrates this combination using sunspot data [2].

The Appendix H example combines signal selection, interval selection, and a final seasonal-strength comparison. We designed this composition to test whether the agent correctly uses both earlier selections in the final analysis.

Practical usefulness. We tested whether this training helps with independently authored analytical questions by adding TSRBench and TimeSeriesExam to the TimeOmni-1 and MTBench evaluations in the original manuscript. We compared RITS with its Qwen3-4B base using the same inputs and tools within each benchmark. Both models chose their own analysis steps, without a supplied workflow or benchmark-specific training.

BenchmarkExamples of analytical objectivesBase accuracy (%)RITS accuracy (%)
TSRBench [3]Calculate turbine utilization; infer directed relationships among measured river records22.7435.61
TimeSeriesExam v1.1 [4, 5]Compare variability and other statistics in synthetic series45.1753.49
TimeOmni-1 [6]Infer upstream–downstream river relationships and identify profit-maximizing actions34.0243.39
MTBench [7]Interpret stock and temperature behavior using reports and observed series43.4158.71

RITS achieved higher accuracy on all four benchmarks while choosing its own analysis steps. To understand where these gains occur, we also examined tasks involving recorded observations, application-oriented calculations, and statistical skills.

Task group or skillBase accuracy (%)RITS accuracy (%)
TSRBench — Public-data reasoning8.9627.83
TSRBench — Numerical reasoning12.0040.25
TimeSeriesExam — Similarity analysis50.8366.67
TimeSeriesExam — Noise understanding53.5765.48

The public-data reasoning tasks include identifying relationships among measured river-gauge series and interpreting recorded events. For a numerical-reasoning example, RITS retrieved the supplied power observations and correctly calculated a wind turbine’s capacity factor, which measures how fully its generating capacity was used. RITS also improved in all five TimeSeriesExam skills, including the similarity and noise tasks shown above.

These results support the usefulness of structured workflow training for independently authored questions with different analytical objectives.

In the revised manuscript, Appendix C.3 explains the template design rationale. Section 4.4 reports external transfer, with the four-benchmark results in Appendix G.6, Table 30, and evaluation details in the same subsection.

AuthorsOpenReview ↗

Response to Reviewer gtHX (2/3)

2. Source coverage and temporal diversity

Each of the 26 “series” is a multivariate source file containing multiple numeric channels. The files come from nine public corpora and cover traffic, electricity, transformer measurements, weather, exchange rates, illness records, and market data.

We have added the requested per-source statistics. Five examples below illustrate differences in record length, sampling interval, dimensionality, and contribution to the question pool.

SourceObservationsCalendar coverageSampling intervalNumeric channelsQuestions generated
PEMS08_flow — traffic [8]17,8562016-07-01 to 2016-08-315 minutes170331
electricity70,0792013-01-01 to 2015-01-0115 minutes320536
ETTh1 — transformer measurements [9]17,4202016-07-01 to 2018-06-261 hour7285
exchange_rate7,5881990-01-01 to 2010-10-10Daily8368
illness8432009 week 40 to 2025 week 48Weekly12284

Appendix C.1, Table 5, provides the complete metadata and generated-question counts for all 26 source files.

We also measured temporal variation in the actual signals used by the questions across all 26 source files. The examples below show differences in linear trends and recurring patterns. Values are medians across the analyzed inputs in each source group.

Linear trend measures the percentage of variation explained by a fitted straight line. Seasonal strength is an STL-based score from 0 to 1, with higher values indicating stronger recurring patterns at the listed period [10].

Data sourceLinear trend (%)Period examinedSeasonal strength (0–1)
PEMS08 traffic0.424 hours0.86
Electricity1.224 hours0.90
Exchange rates41.27 days0.30
Illness records2.952 weeks, approximately one year0.85

Traffic and electricity show pronounced daily repetition. Exchange-rate inputs have stronger linear trends and weaker seven-day repetition, while illness inputs show pronounced annual repetition. These measurements demonstrate variation in the underlying signals used for the analytical tasks. Appendix C.2 and Table 6 provide the measurement definitions, calculation details, and temporal-property results.

3. Scoring valid execution orders

Thank you for pointing this out. Both execution orders in your example are valid and should receive the same score.

Your example has two branches after data loading, r → a → c and r → b. Once a finishes, c can run before or after b. Therefore, both (r, a, b, c) and (r, a, c, b) satisfy the dependencies.

The original algorithm matched a and b together before considering c. For (r, a, c, b), it then searched for c only after b, missing the valid c call that had already occurred. This could incorrectly lower the score.

We checked all 101 supported templates and found six that could be affected by this scoring issue. We revised Algorithm 1 to match calls across the whole workflow while respecting their dependencies. Independent calls can appear in either order, so both valid orders in your example receive the same score.

We then recomputed the scores for all eight main models using the revised matching rule. The Trace rankings remain unchanged on both ID and OOD. All entries below show original → corrected scores, with RITS averaged over three training runs.

ModelID Trace (%)OOD Trace (%)
GPT-5.6-sol92.79 → 92.8092.75 → 93.21
Gemini 3.1 Pro90.32 → 91.0190.91 → 91.44
Claude Opus 4.891.00 → 91.4089.97 → 91.33
Qwen3-4B (zero-shot)40.41 → 40.6844.88 → 45.23
Ministral-3-8B-Reasoning78.44 → 80.8083.63 → 85.60
Qwen3.6-27B86.35 → 86.7288.83 → 89.14
SFT88.03 → 88.3294.57 → 94.66
RITS98.72 → 98.7593.97 → 94.64

RITS’s Trace-EM and OV-F1 also remain unchanged.

MetricID (%)OOD (%)
Trace-EM92.00 → 92.0063.07 → 63.07
OV-F198.33 → 98.3389.32 → 89.32

The revised matching rule gave some responses more partial credit, increasing Trace. Those responses still differed from the reference workflow, so they did not qualify as complete matches under Trace-EM. The number of correctly matched tool names also stayed the same, leaving OV-F1 unchanged.

In the revised manuscript, Algorithm 1 in Appendix F gives the revised matching procedure and explains both valid execution orders. Table 1 reports the revised Trace scores, and Appendix G.8, Table 34, reports Trace-EM and OV-F1 for all eight models.

AuthorsOpenReview ↗

Response to Reviewer gtHX (3/3)

4. Added value of trace alignment beyond intermediate supervision

Trace alignment adds explicit supervision for using the requested tools and arguments throughout the workflow. A correct intermediate value can also arise from an incorrect analysis step. For example, in one evaluated case, the agent analyzed the wrong time interval but obtained the expected intermediate value. Trace alignment additionally checks the interval argument, providing feedback on whether the requested analysis was followed.

Independent training runs with retrieved tools

To evaluate this additional value, we compared R3, which uses answer, format, and intermediate-value rewards, with RITS, which also uses the revised trace-alignment reward.

We evaluated two independently trained models per variant on the same questions and retrieved tool candidates. Each model was evaluated over three inference runs at temperature 0.6. The table reports the mean and standard deviation across training runs, after averaging the inference runs for each model.

We measure complete workflow agreement using Trace-EM, which requires every reference tool call and required argument value to match in a valid dependency order, with no extra calls.

SplitMethodTrace-EM (%)Accuracy (%)
IDR386.97 ± 0.7180.40 ± 1.23
IDRITS89.07 ± 0.6681.27 ± 1.23
OODR360.70 ± 2.7865.67 ± 0.38
OODRITS62.43 ± 2.5067.23 ± 1.93

RITS achieves higher mean Trace-EM on both splits, by 2.10 percentage points on ID and 1.73 points on OOD. Mean answer accuracy also increases. These independent training replications support an additional benefit in matching the complete requested workflow, including tool selection and arguments.

Comparison with a simpler reward

We also evaluated the execution-success-plus-call-cost reward you suggested. Simple retains the answer and format rewards and adds a reward for successful tool execution and a penalty for every tool call. Execution success means that the call returns without an error. This objective introduces a trade-off between completing the analysis and reducing tool use.

The following comparison uses greedy decoding with retrieved tools.

Reward settingID accuracy (%)OOD accuracy (%)
Simple43.630.6
RITS78.465.6

RITS achieves 34.8 percentage points higher accuracy on ID and 35.0 points higher on OOD.

SplitHopsSimple answer rate (%)RITS answer rate (%)
ID2–396.1296.12
ID4–51.4997.76
OOD2–383.5196.81
OOD4–50.6487.50

Simple fails to complete answers on most high-hop questions. Its reward penalizes every tool call, including calls required to complete the workflow. This pattern is consistent with pressure to reduce tool use without reference-guided procedure supervision. RITS rewards correctly matched required calls and reduces trace agreement for unmatched extras.

This comparison supports the value of reference-guided procedure supervision.

Together, the two comparisons clarify the contribution of trace alignment. The Simple comparison supports supervising the analytical procedure using reference workflows. The repeated R3 comparison provides additional evidence that trace alignment improves complete workflow agreement when intermediate supervision is already present.

In the revised manuscript, Section 4.3 discusses the reward components. Appendix G.3.1 reports the independent training comparison in Table 20, OV-F1 in Table 21, the greedy comparison in Table 22, and the Simple comparison in Table 23.

References

[1] Yazdanian et al., On the radar: Predicting near-future surges in skills’ hiring demand to provide early warning to educators, Computers and Education: Artificial Intelligence 3, 100043, 2022.

[2] statsmodels official example notebook, Stationarity and detrending (ADF/KPSS).

[3] Yu et al., TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models, ICML 2026.

[4] Cai et al., TimeSeriesExam: A Time Series Understanding Exam, NeurIPS Workshop on Time Series in the Age of Large Models 2024.

[5] AutonLab, TimeSeriesExam v1.1, Hugging Face dataset release, 2025.

[6] Guan et al., TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models, ICLR 2026.

[7] Chen et al., MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering, arXiv:2503.16858, 2025.

[8] Guo et al., Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting, AAAI 2019.

[9] Zhou et al., Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI 2021.

[10] Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts, 2021.

Reviewer_hCSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies time-series analysis agents, focusing on natural-language questions that require a multi-step workflow of analysis tools to answer. The authors argue that existing data for this setting is unreliable for both training and evaluating agents. To address this, they propose RITS, a framework for constructing trace-identifiable multi-hop time-series questions that can then be used for agent training and evaluation. RITS structures question templates as directed acyclic graphs (DAGs) over a library of 308 parameterized tools. The authors further post-train a tool-using agent with a trace-aware reward. Experimental results demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The paper is well organized and easy to follow. The illustrations are vivid and easy to understand.
  2. Simple and intuitive approach. The proposed method is straightforward. The RITS framework is intuitive and easy to understand, which makes it easy to reproduce and build upon.
  3. Reasonable contributions and broad baseline comparison. The paper makes several concrete contributions, including the structured DAG-based question templates, the resulting dataset, and a tool-using agent post-trained on this dataset. The set of baselines is fairly comprehensive, and the experimental results offer useful reference value for future research on time-series analysis agents.
Critical weaknesses
  1. Strong data assumptions and limited evidence of generalization. Both the method and the experiments rely entirely on data constructed by RITS, which imposes strong requirements on the data, such as the availability of DAG-structured templates and parameterized tools. It is therefore unclear whether the approach can generalize to other datasets and application scenarios. Although the paper includes some generalization experiments, they are rather limited, covering only a few datasets and a small number of tasks. A more extensive evaluation on diverse, externally sourced time-series benchmarks would substantially strengthen the paper's claims.
  2. Weak out-of-distribution performance of the post-trained agent. The post-trained model falls considerably behind other models in several settings. In particular, its accuracy on the OOD split and on unseen templates lags behind Claude by tens of percentage points. This suggests that the post-trained model generalizes poorly, and raises the concern that it may only work well when the test data follows the same distribution as the training data (i.e., in the ID setting). The authors should discuss this gap and clarify to what extent the gains from post-training transfer beyond the training distribution.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer hCSr (1/2)

Thank you for recognizing the simple, easy-to-understand design of RITS. We also appreciate your positive assessment of the dataset and trained agent, and the broad comparison with baseline models. Below, we present the expanded external benchmark results, examine how post-training improves OOD performance, and analyze the remaining gap to Claude.

1. Generalization beyond RITS-constructed data

To address your concern about generalization to other datasets and application scenarios, we added evaluations on TSRBench [1] and TimeSeriesExam [2, 3] alongside the TimeOmni-1 [4] and MTBench [5] evaluations reported in the original manuscript. TSRBench covers 15 tasks across 14 application domains, including questions grounded in public records and application-motivated simulations. TimeSeriesExam evaluates five statistical skills on synthetic time series.

We compared RITS with its tool-enabled Qwen3-4B base using the same inputs and tool access within each benchmark, without benchmark-specific fine-tuning. Both agents chose their own analysis steps without a supplied reference workflow.

BenchmarkExamples of analytical objectivesBase accuracy (%)RITS accuracy (%)
TSRBench [1]Calculate turbine utilization; infer directed relationships among measured river records22.7435.61
TimeSeriesExam v1.1 [2, 3]Compare variability and other statistics in synthetic series45.1753.49
TimeOmni-1 [4]Infer upstream–downstream river relationships and identify profit-maximizing actions34.0243.39
MTBench [5]Interpret stock and temperature behavior using reports and observed series43.4158.71

RITS achieves higher accuracy on all four benchmarks, showing that the benefits of training extend beyond questions generated by RITS. To understand where these gains occur, we also examined performance on analytical tasks involving recorded observations, application-oriented calculations, and statistical analysis. Selected results are shown below.

Task group or skillBase accuracy (%)RITS accuracy (%)
TSRBench — Public-data reasoning8.9627.83
TSRBench — Numerical reasoning12.0040.25
TimeSeriesExam — Similarity analysis50.8366.67
TimeSeriesExam — Noise understanding53.5765.48

The public-data reasoning questions include identifying relationships among measured river-gauge series and interpreting basketball records and time-stamped events. The improvement on these questions provides evidence of transfer to analysis grounded in recorded observations.

The numerical-reasoning questions have explicit application objectives. For example, a wind-energy question asks for capacity factor to assess how fully a turbine’s generating capacity was used. RITS correctly used the supplied power observations to answer it. This illustrates using numerical evidence to meet an analytical objective when the question supplies no analysis procedure. RITS also improves in all five TimeSeriesExam skills, including the similarity and noise tasks shown above.

Together, these results support the usefulness of controlled workflow training for independently authored analytical questions across different datasets and application contexts. In the revised manuscript, Section 4.4 reports the expanded external evaluations. Appendix G.6, Table 30, summarizes the four-benchmark results. Table 31 gives the evaluated scopes, and the same subsection describes the evaluation settings.

AuthorsOpenReview ↗

Response to Reviewer hCSr (2/2)

2. OOD performance and the gap to Claude

Post-training improves overall OOD accuracy from 34.4% to 69.5%, with gains on both OOD splits. The unseen template split tests specific templates and combinations of operations excluded from training. The unseen tool split tests selected tools or combinations of parameter values excluded from training.

SystemOverall OOD accuracy (%)Unseen template (%)Unseen tool (%)
Qwen3-4B Base34.438.830.0
RITS69.560.878.1
GPT-5.6-sol67.458.876.0
Gemini 3.1 Pro67.860.874.8
Claude Opus 4.884.691.677.6

RITS gains 22.0 percentage points on the unseen template split and 48.1 points on the unseen tool split, providing evidence that the learned skills transfer beyond the training distribution. RITS matches or slightly exceeds GPT and Gemini on both splits and performs close to Claude on the unseen tool split. The substantial remaining gap to Claude occurs on the unseen template split.

To better understand why Claude outperformed RITS on the unseen template split, we examined the results by task type. Most of the performance gap came from just two tasks, estimating the timing difference between two signals (phase lag) and comparing how strongly two signals are related (peak coherence strength). On the other tasks in the unseen template split, the two models performed similarly.

TaskRITS accuracy (%)Claude accuracy (%)
Estimating timing differences between signals (phase lag)11.43100.00
Comparing the strength of signal relationships (peak coherence)35.29100.00
Other tasks in the unseen template split86.5087.12

Why did RITS struggle with phase lag? The main issue was choosing between two similar tools. The correct tool, phase_lag_at_peak_coherence, measures the timing difference at the frequency where two signals are most closely related. RITS often selected phase_lag_at_frequency_welch, which uses a different rule to choose the frequency. Because the two tools can examine different frequencies, they can return different answers. Claude consistently chose the correct tool.

We tested whether clearer tool names could help RITS make the right choice. We renamed the competing tool to phase_lag_at_peak_cross_spectrum_magnitude to make its calculation method more explicit. Everything else, including the tool descriptions and required inputs, remained unchanged.

Tool namingBase accuracy (%)RITS accuracy (%)
Original name10.0010.00
More descriptive name20.0087.14

Simply changing the tool name dramatically improved performance. RITS’s final-answer accuracy increased from 10.00% to 87.14%, without any retraining.

These results identify difficulty distinguishing between similar tools as a major source of error in this task. They also show that clearer tool names can help RITS choose the intended operation, consistent with findings from previous studies [6, 7].

Together, these results demonstrate substantial gains beyond ID and identify specific unseen analyses where difficulties remain. The comparison with Base establishes the benefit of post-training on both OOD splits, and the task-level analysis explains where Claude’s advantage is concentrated and provides a concrete diagnosis for one of those tasks.

In the revised manuscript, Table 1 reports the OOD comparison, and Section 4.2 discusses the post-training gains on unseen templates and unseen tools. The task-level diagnosis and tool-naming comparison are provided in this response.

References

[1] Yu et al., TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models, ICML 2026.

[2] Cai et al., TimeSeriesExam: A Time Series Understanding Exam, NeurIPS Workshop on Time Series in the Age of Large Models 2024.

[3] AutonLab, TimeSeriesExam v1.1, Hugging Face dataset release, 2025.

[4] Guan et al., TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models, ICLR 2026.

[5] Chen et al., MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering, arXiv:2503.16858, 2025.

[6] Lee et al., Don’t Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models, ACL 2026.

[7] Ye et al., RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning, EMNLP 2024.

We would like to thank you again for your thoughtful feedback. As the discussion period is coming to an end, please let us know if you have any remaining questions or concerns. We would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.