- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes TSHarness, an agentic framework for cross-dataset zero-shot time-series question answering (TSQA). The method combines numerical tools, a structured Time-Series Perception State (TPS), an evidence-request mechanism, procedural memory, and a reinforcement-learning-based Tool Selector. A Perceiver calls numerical tools to fill a structured Time-Series Perception State (TPS), and an Answerer reasons over the TPS and can request more evidence. An RL-trained Tool Selector prunes the Perceiver’s tool plan. All components are built on TSRBench and evaluated frozen on five target benchmarks with Qwen3.8-27B and GPT-5.6-sol backbones.
Critical strengths
- The system is built on one source benchmark, all components are frozen before target evaluation, and transfer is tested on five targets with different input series, question types, and answer formats, under two backbones.
- The paper presents a useful tool-augmented system with zero-shot transfer results.
Critical weaknesses
-
The paper attributes the transfer gap to entangling numerical perception with answer generation, but the only support is Figure 1(a). That figure compares a few older specialized models, each with its own base model, training recipe, and output behavior. A performance gap across such different models shows that transfer is hard. It does not establish that entanglement is the cause, or that the gap comes from the two proposed shifts. The shifts themselves are never quantified across the benchmarks. The method is also not explicitly mapped to the problem. One natural reading is that the Perceiver and its tools address perceptual shift, and the Answerer addresses contextual shift. The paper never states this mapping or tests it. Table 2 ablates tools and memory, but it does not show which shift each component mitigates, or whether separating the Perceiver from the Answerer matters on its own. The authors should make this connection explicit: which component targets which shift, and what evidence shows it does.
-
The claim of “remarkable” gains is not supported by the evidence. The introduction states that TSHarness “remarkably outperforms same-backbone baselines” across five targets, but the margins are uneven. With Qwen3.8-27B, TSHarness trails TimeClaw on CaTSBench (−0.24) and leads it on TimeOmni by 0.49 points, two questions out of 412. With GPT-5.6-sol, it is compared only against direct answering. It lowers accuracy on TimeOmni, and its gains on CaTSBench (+0.48) and TSAQA (+0.73) are a few questions each. Clear gains appear on only a subset of targets, mainly ChatTime and TSAQA with Qwen. The paper also does not report the number of runs, error bars, or significance tests for Table 1 or Table 2. Results appear to come from a single run at temperature 0, and a separate run of the no-selector harness on TSRBench gives 61.89 against the 62.62 in Table 1 (App. C.5). Variation of this size is comparable to several reported margins. The authors should report results over multiple runs with confidence intervals or paired significance tests, and qualify the claim accordingly.
-
The baseline comparison is limited. The agentic baselines are CoT, ReAct, and TimeClaw on Qwen only. The GPT backbone is compared against zero-shot answering alone. Methods the paper discusses in related work are absent: TS-Agent, AION, and KairosAgent among harnesses, and ChatTS and TimeOmni-1 among trained models. The tool set given to ReAct and the TimeClaw configuration under the frozen-source protocol are also not specified.
-
Tools are the main source of the gains, yet the paper does not say clearly who writes them or how. App. B.1 mentions “implementing new operations” and activation rules, but does not specify whether humans or an LLM implement them, or how many tools were added during source development. The paper also does not say how the tool families were chosen.
-
The abstract describes TSHarness as a “cost-efficient foundation,” but each question passes through five LLM roles (Table 6). The TSRBench selector comparison records about 8.2 model calls per question (3,393 over 412 questions; App. D.3), compared with a single direct answer for the zero-shot baseline. The paper reports no token, latency, or dollar costs, and includes no compute-matched baseline, such as self-consistency or ReAct under the same call budget. Part of the improvement may therefore come from additional test-time compute rather than from the proposed design.
Questions for authors
-
Have you compared against a single-agent baseline, such as ReAct, given the identical tool library and budget? Without it, how do you separate the benefit of the Perceiver/Answerer split from the benefit of tool access? Table 2 shows the pipeline without tools averages below direct zero-shot answering on the targets.
-
Is there an error analysis that attributes failures to mis-perception of the series versus misinterpretation of the question or answer format? Can you quantify the perceptual shifts (sampling rate, length, missingness, channels) across the six benchmarks?
-
Which components differ per target benchmark: adapters, parsers, evidence-requirement rules, task extensions? Who designed each, and did anyone involved inspect the target questions?
-
Were the tools implemented by humans or by an LLM? How many were added during source development, and how were the tool families chosen?
-
Which tools did the ReAct baseline have access to? Was TimeClaw's memory and capability evolution built on TSRBench and frozen, or allowed to update on the targets?
-
Removing Memory slightly raises average target accuracy (Table 2). In what sense is Memory reusable across benchmarks?
-
Several ChatTime accuracies in Tables 1, 2, and 10 (e.g., 96.10, 74.13) are not multiples of 1/412. Which question set were these computed on? How many runs per method were excluded as incomplete under Eq. (8)?
4 comments
W3. Coverage and reproducibility of the baseline comparisons.
TSHarness is an agentic framework, so our primary comparisons focus on reasoning and tool-use workflows: CoT as a reasoning baseline, ReAct as a general tool-using agent, and TimeClaw as a closely related time-series agentic method. Additional matched-tool comparisons with ReAct and TimeClaw assess workflow contributions using the same backbone and shared numerical tool library.
The other named methods differ in scope and implementation availability:
- AION evaluates complete time-series project workflows, including model development and validation. KairosAgent focuses on forecasting and future-morphology reasoning. Applying either to our per-question TSQA setting requires changes to its task interface and evaluation protocol.
- For TS-Agent, we could not locate an official implementation of the cited paper. An independent reproduction introduces implementation choices that affect comparison fidelity.
- ChatTS and TimeOmni-1 are relevant specialized models. Our evaluation prioritizes agentic comparisons under the same backbone, with ChatTime and Time-MQA providing representative trained-model comparisons. The comparison is therefore representative rather than exhaustive.
W4 & Q4. Tool-library construction and the role of human engineering.
Tool implementation, tool configuration, and activation-rule development are distinct parts of resource construction.
The base tool library is specified once by humans, covering standard statistical and temporal properties, including descriptive statistics, value distributions, trend, seasonality, volatility, anomalies, change points, segmentation, and cross-series relationships.
During source construction, additional reusable tools were generated using GPT-5.6-sol, consistent with the general approach to LLM-based tool creation explored in LATM [1] and CRAFT [2]. Their implementations were executed, checked on source-development examples, and integrated through the shared tool interface. This stage five specialized tool candidates and introduced two additional diagnostic tools: candidate-trajectory comparison and event-impact interpretation, motivated by analytical needs identified from source execution traces and answer feedback.
The resulting library is frozen and reused across target benchmarks without target-specific tool development. We clarify the construction procedure and tool inventory in Appendices B.1 and B.2.
Reference
[1] Cai et al. Large Language Models as Tool Makers. ICLR 2024.
[2] Yuan et al. CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets. ICLR 2024.
W5. End-to-end efficiency and the contribution of additional test-time compute.
We added total input and output tokens per question to the revised evaluation, alongside accuracy.
Main comparison (Table 1): averages across five target benchmarks.
| Method | Avg. accuracy (%) ↑ | Tokens per question (k) ↓ |
|---|---|---|
| ReAct | 64.66 | 17.90 |
| TimeClaw | 66.79 | 28.75 |
| TSHarness | 74.95 | 15.82 |
The revised Main Results section explicitly discusses this accuracy–cost comparison: TSHarness achieves higher average accuracy with 11.6% and 45.0% fewer tokens than ReAct and TimeClaw, respectively. This provides evidence about LLM inference consumption beyond the original tool-execution counts.
Shared-tool comparison (Table 5): averages across six datasets, using 128 sampled questions per dataset and shared numerical tools.
| Method | Avg. accuracy (%) ↑ | Tokens per question (k) ↓ |
|---|---|---|
| ReAct | 65.49 | 22.73 |
| TimeClaw | 52.73 | 38.10 |
| TSHarness | 72.27 | 17.63 |
With the same backbone and shared numerical tools, TSHarness achieves higher average accuracy while consuming fewer tokens than ReAct and TimeClaw. These results demonstrate better end-to-end performance and token efficiency in this comparison.
Together, these experiments support a better accuracy–token-cost trade-off relative to the evaluated agentic baselines. Direct zero-shot answering remains cheaper, and token consumption does not by itself establish lower latency or monetary cost.
Q1. Does the Perceiver/Answerer split help when a single agent has identical tools and budget?
(1) Shared-tool comparison with single-agent ReAct (Table 5).
We added a comparison using the same Qwen3.8-27B backbone and shared numerical tools, evaluated on 128 randomly sampled questions per dataset across six datasets.
| Method | Avg. accuracy (%) ↑ | Tokens per question (k) ↓ |
|---|---|---|
| ReAct | 65.49 | 22.73 |
| TimeClaw | 52.73 | 38.10 |
| TSHarness | 72.27 | 17.63 |
With the same backbone and shared numerical tools, TSHarness achieves higher average accuracy while consuming fewer tokens than ReAct and TimeClaw. This supports a contribution from pipeline organization beyond numerical-tool access alone.
(2) Evidence organization (Table 3).
We additionally examine evidence organization, individual evidence fields, and feedback. All other listed TPS components are retained unless explicitly removed below.
| Case | Configuration | Evidence loop | TSRBench: Qwen | TSRBench: GPT | Time-MQA: Qwen | Time-MQA: GPT | ChatTime: Qwen | ChatTime: GPT |
|---|---|---|---|---|---|---|---|---|
| A1 | Without TPS structure | ✗ | 59.38 | 71.88 | 48.44 | 53.12 | 96.84 | 97.33 |
| A2 | Structured TPS with all evidence fields | ✗ | 58.59 | 71.88 | 51.56 | 57.03 | 96.60 | 97.57 |
| B1 | Without statistics | ✗ | 37.50 | 44.53 | 47.66 | 46.88 | 96.84 | 95.15 |
| B2 | Without temporal features | ✗ | 38.28 | 46.09 | 54.69 | 51.56 | 42.48 | 42.48 |
| B3 | Without textual summaries | ✗ | 58.59 | 71.09 | 51.56 | 53.12 | 96.36 | 97.57 |
| B4 | Without task extensions | ✗ | 35.94 | 52.34 | 49.22 | 56.25 | 97.33 | 88.11 |
| C1 | Without evidence requirements | ✗ | 58.59 | 71.88 | 52.34 | 56.25 | 96.36 | 97.57 |
| C2 | Without evidence requirements | ✓ | 57.03 | 72.66 | 53.91 | 56.25 | 97.09 | 98.06 |
| Full | Full TPS and feedback | ✓ | 57.81 | 72.66 | 54.69 | 57.03 | 96.84 | 98.30 |
The revised TPS ablation discussion distinguishes three findings:
-
Evidence organization contributes beyond the presence of evidence fields.
-
Different tasks depend on different evidence.
-
Feedback provides selective benefits.
(3) Dependence on numerical tools (Tables 1–2).
The reviewer correctly observes that removing tools lowers average target accuracy below direct zero-shot answering. The averages across the five targets are:
| Configuration | Avg. target accuracy (%) |
|---|---|
| Direct zero-shot | 65.44 |
| TSHarness without tools | 63.06 |
| Full TSHarness | 74.95 |
As discussed in the revised Tools and Memory analysis, numerical tools are a key contributor. Removing them tests the workflow’s dependence on numerical evidence.
The added shared-tool comparison addresses that resource-access concern, while the TPS ablations examine evidence organization and acquisition. Together, these results support the complete tool-grounded workflow.
Q2. Can failures and benchmark shifts be quantified separately?
(1) Separating missing-evidence failures from downstream answering.
The added Required evidence removal and recovery analysis (Figure 5) removes required TPS evidence on ChatTime while keeping the readout fixed:
| Evidence condition | Qwen3.8-27B accuracy (%) | GPT-5.6-sol accuracy (%) |
|---|---|---|
| Full evidence | 90.29 | 96.36 |
| Required evidence removed | 36.89 | 41.02 |
| After targeted evidence recovery | 90.29 | 96.36 |
With the evidence gaps explicitly specified, targeted re-perception restores both scores to the full-evidence level. This isolates the contribution of missing numerical evidence. Figure 4 additionally tracks corrections and regressions after evidence integration.
(2) Quantifying input differences across the six benchmarks.
We analyzed the numerical inputs in the current evaluation benchmark:
| Benchmark | Length: median [P25, P75] | Length range | Series per question: median / max | Channels per episode: max |
|---|---|---|---|---|
| TSRBench | 128 [35, 1000] | 24–1024 | 4 / 12 | 12 |
| Time-MQA | 24 [24, 24] | 3–24 | 1 / 5 | 1 |
| ChatTime | 128 [64, 512] | 64–512 | 1 / 1 | 1 |
| CaTSBench | 24 [7, 74] | 3–150 | 2 / 4 | 1 |
| TSAQA | 55 [35, 111] | 6–343 | 1 / 5 | 1 |
| TimeOmni-1 | 62 [60, 62] | 14–744 | 1 / 2 | 1 |
The subsets differ in sequence length and input dimensionality. Length statistics summarize available scalar sequences, and series counts are computed per question with numerical input. Independent candidate series are distinguished from aligned multivariate channels. These statistics characterize the evaluated subsets. We add this table and its definitionto Appendix A.3.
The benchmarks also differ in the task capabilities they assess. The added Figure 3 compares CoT, ReAct, TimeClaw, and TSHarness across 26 tasks from the six benchmarks, showing that the methods have task-specific strengths and weaknesses.
Q3. Which components differ per target benchmark? Who designed each, and did anyone involved inspect the target questions?
(1) How the architecture accommodates different target benchmarks.
- Adapters and parsers translate benchmark-specific inputs and answer formats into a shared interface.
- Evidence-requirement rules map the question and task description to the required evidence types.
- Task extensions and the Answerer adapt the analysis and interpretation to the current question, while retaining the shared TPS structure and registered operations.
(2) Design responsibility and development information.
- The adapters, parsers, fixed mappings, and extension interface were implemented during human-directed development. The base LLM in the pipeline generates question-specific analysis requests at inference.
- Target-specific adaptation is limited to implementing adapters and parsers for the published input and answer formats. The source-developed tools and Memory are reused, and all implementations remain fixed during target evaluation.
The added Appendix A.2 clarifies this interface adaptation and the frozen evaluation protocol.
Q5. What tools did ReAct receive, and how was TimeClaw’s evolution controlled?
(1) ReAct’s tool library.
ReAct uses an 11-tool library: channel listing, series overview, channel statistics, value retrieval, autocorrelation, periodicity detection, peak detection, ARIMA forecasting, portfolio VaR, Sharpe ratio, and CAPM regression.
(2) TimeClaw’s source construction and frozen evaluation.
We use TimeClaw’s released implementation. Its episodic Memory is constructed from trajectories generated on TSRBench construction examples, excluding the held-out test split. Target evaluation uses the frozen Memory bank and a fixed tool library, with fingerprint-based retrieval. No target-side Memory updates, tool modifications, or feedback-driven capability evolution are performed. Target reference answers are used only for scoring. This follows the same source-only development and frozen-transfer setting as TSHarness.
(3) Additional comparison with the same tool library.
We give ReAct, TimeClaw, and TSHarness the same numerical-tool library and Qwen3.8-27B backbone. All methods answered the same 128 randomly sampled questions per dataset (Table 5).
| Method | Avg. accuracy (%) | Tokens/question (k) | TSRBench | Time-MQA | ChatTime | CaTSBench | TSAQA | TimeOmni |
|---|---|---|---|---|---|---|---|---|
| ReAct | 65.49 | 22.73 | 47.66 | 60.94 | 93.75 | 87.50 | 54.69 | 48.44 |
| TimeClaw | 52.73 | 38.10 | 39.06 | 53.91 | 81.25 | 88.28 | 32.03 | 21.88 |
| TSHarness | 72.27 | 17.63 | 60.94 | 65.63 | 98.44 | 98.44 | 53.91 | 56.25 |
These results show that TSHarness’s average advantage persists under shared tool access.
Q6. What does cross-benchmark reusability of Memory mean when removing it improves average accuracy?
Memory stores procedural guidance developed from TSRBench construction examples, including applicability conditions and recommended analyses. It helps the Perceiver select appropriate tools and analytical procedures. Table 2 shows that removing Memory lowers TSRBench accuracy, supporting its usefulness on the source benchmark.
Reusability refers to deploying this source-developed guidance across benchmarks without target-specific updates. Its benefit depends on whether the guidance matches the target task, so reuse does not guarantee improvement on every benchmark.
We acknowledge this limitation in Conclusion and Limitations: “Procedural memory mainly benefits in-domain performance, while its source-derived priors provide limited support for zero-shot transfer.” Improving Memory’s transferability is identified as an important direction.
Q7. Clarifying evaluation, accuracy computation.
We acknowledge this mistake for agentic methods tested on ChatTime (e.g., 96.10% and 74.13%), which were computed on wrong 4000 questions evaluation subset. We have updated the ChatTime evaluation to use the same fixed 412-question subset across the methods in the main agentic comparison and updated the corresponding results. Besides, We conducted three independent runs for TSHarness and TSHarness* and report their mean accuracies, consistent with the repeated-run evaluation described in W2.
Incomplete executions and scoring. Each main-table agentic run is evaluated on 412 questions. Accuracy is the number of correct answers divided by 412. Missing, invalid, or unparseable answers count as incorrect, and no questions are excluded from the denominator.
We have updated Appendix A.4 (Eq. 8) to clarify the denominator and how these cases are scored.
(1) Motivation and component mapping.
Figure 1(a) illustrates the cross-dataset transfer gap. Our discussion of perceptual and contextual shifts motivates separating reusable numerical measurements from question-dependent interpretation. The revised experiments examine this design through TPS and evidence-loop ablations and comparisons with ReAct and TimeClaw using shared numerical tools. The components address these challenges as follows:
(2) Evidence in the revised paper.
Together, these analyses connect the proposed components to their roles in evidence acquisition and contextual answering, while distinguishing the design motivation from the empirical evidence.
(1) Higher average accuracy.
We revised the contribution statement in the Introduction from “It remarkably outperforms same-backbone baselines without any target-side adaptation.” to “It outperforms same-backbone baselines in average target accuracy without target-side training or answer feedback.”
This makes the claim explicitly about average transfer performance. The expanded Table 1 includes an Avg. column, and added Figure 3 compares CoT, ReAct, TimeClaw, and TSHarness across 26 tasks in 6 datasets. These comparisons reveal task-specific strengths across the evaluated methods: relative rankings vary across capabilities and datasets, and gains can be large, small, or negative.
Section 4.2 explicitly acknowledges this: “The capability profiles show that aggregate transfer gains coexist with task-specific strengths and weaknesses.” We also discuss TSHarness outperforming TimeClaw on TSRBench reasoning while trailing it on CaTSBench retrieval.
Across the five target benchmarks:
The supported conclusion is therefore higher average accuracy across heterogeneous targets, with explicitly acknowledged differences across tasks.
(2) Multiple runs.
Reporting main results without run-level standard deviations or confidence intervals also occurs in prior work, including TimeClaw, ChatTime, Time-MQA, and TimeOmni-1. Nevertheless, we agree that the repeated evaluation strengthens the reliability of the comparison.
Section 4.1 of the revised manuscript specifies three-run means for the reported TSHarness and TSHarness* configurations. Table 1 presents the updated averages. We also update the relevant experimental descriptions and results in the appendix to keep them consistent with the revised main text. GPU availability and the rebuttal time limit restricted repeated evaluation of every baseline and ablation. Complete multi-run coverage is planned for a subsequent manuscript update. Small dataset-level margins are not presented as statistically significant improvements.