acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Time-Series Question Answering via Decoupled Perception and Reasoning

19 views

Abstract

Time-series question answering (TSQA) requires grounding linguistic queries and diverse answer formats in complex numerical observations. However, existing methods heavily overfit to specific datasets and struggle to generalize when input series, question contexts, and answer requirements shift simultaneously. To address this challenge, we propose TSHarness, an agentic framework that establishes a decoupled workflow for cross-dataset zero-shot TSQA. At its core, TSHarness divides and conquers numerical perception and contextual reasoning via a structured Time-Series Perception State (TPS). Guided by a reusable memory of analytical knowledge, a learned Tool Selector adaptively invokes numerical tools to extract salient statistical and temporal features into the TPS. The answering agent then performs semantic reasoning over the TPS to generate target outputs, triggering iterative re-perception through the feedback loop when evidence is deemed insufficient. By separating numerical feature extraction from question-specific reasoning, TSHarness eliminates the need for target-side training or answer feedback, providing a generalizable, cost-efficient foundation for zero-shot TSQA.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series Question Answering, Generalizable Agentic Harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tMfwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes TSHarness, an agentic framework for cross-dataset zero-shot time-series question answering (TSQA). The method combines numerical tools, a structured Time-Series Perception State (TPS), an evidence-request mechanism, procedural memory, and a reinforcement-learning-based Tool Selector. A Perceiver calls numerical tools to fill a structured Time-Series Perception State (TPS), and an Answerer reasons over the TPS and can request more evidence. An RL-trained Tool Selector prunes the Perceiver’s tool plan. All components are built on TSRBench and evaluated frozen on five target benchmarks with Qwen3.8-27B and GPT-5.6-sol backbones.

Critical strengths
  1. The system is built on one source benchmark, all components are frozen before target evaluation, and transfer is tested on five targets with different input series, question types, and answer formats, under two backbones.
  2. The paper presents a useful tool-augmented system with zero-shot transfer results.
Critical weaknesses
  1. The paper attributes the transfer gap to entangling numerical perception with answer generation, but the only support is Figure 1(a). That figure compares a few older specialized models, each with its own base model, training recipe, and output behavior. A performance gap across such different models shows that transfer is hard. It does not establish that entanglement is the cause, or that the gap comes from the two proposed shifts. The shifts themselves are never quantified across the benchmarks. The method is also not explicitly mapped to the problem. One natural reading is that the Perceiver and its tools address perceptual shift, and the Answerer addresses contextual shift. The paper never states this mapping or tests it. Table 2 ablates tools and memory, but it does not show which shift each component mitigates, or whether separating the Perceiver from the Answerer matters on its own. The authors should make this connection explicit: which component targets which shift, and what evidence shows it does.

  2. The claim of “remarkable” gains is not supported by the evidence. The introduction states that TSHarness “remarkably outperforms same-backbone baselines” across five targets, but the margins are uneven. With Qwen3.8-27B, TSHarness trails TimeClaw on CaTSBench (−0.24) and leads it on TimeOmni by 0.49 points, two questions out of 412. With GPT-5.6-sol, it is compared only against direct answering. It lowers accuracy on TimeOmni, and its gains on CaTSBench (+0.48) and TSAQA (+0.73) are a few questions each. Clear gains appear on only a subset of targets, mainly ChatTime and TSAQA with Qwen. The paper also does not report the number of runs, error bars, or significance tests for Table 1 or Table 2. Results appear to come from a single run at temperature 0, and a separate run of the no-selector harness on TSRBench gives 61.89 against the 62.62 in Table 1 (App. C.5). Variation of this size is comparable to several reported margins. The authors should report results over multiple runs with confidence intervals or paired significance tests, and qualify the claim accordingly.

  3. The baseline comparison is limited. The agentic baselines are CoT, ReAct, and TimeClaw on Qwen only. The GPT backbone is compared against zero-shot answering alone. Methods the paper discusses in related work are absent: TS-Agent, AION, and KairosAgent among harnesses, and ChatTS and TimeOmni-1 among trained models. The tool set given to ReAct and the TimeClaw configuration under the frozen-source protocol are also not specified.

  4. Tools are the main source of the gains, yet the paper does not say clearly who writes them or how. App. B.1 mentions “implementing new operations” and activation rules, but does not specify whether humans or an LLM implement them, or how many tools were added during source development. The paper also does not say how the tool families were chosen.

  5. The abstract describes TSHarness as a “cost-efficient foundation,” but each question passes through five LLM roles (Table 6). The TSRBench selector comparison records about 8.2 model calls per question (3,393 over 412 questions; App. D.3), compared with a single direct answer for the zero-shot baseline. The paper reports no token, latency, or dollar costs, and includes no compute-matched baseline, such as self-consistency or ReAct under the same call budget. Part of the improvement may therefore come from additional test-time compute rather than from the proposed design.

Questions for authors
  1. Have you compared against a single-agent baseline, such as ReAct, given the identical tool library and budget? Without it, how do you separate the benefit of the Perceiver/Answerer split from the benefit of tool access? Table 2 shows the pipeline without tools averages below direct zero-shot answering on the targets.

  2. Is there an error analysis that attributes failures to mis-perception of the series versus misinterpretation of the question or answer format? Can you quantify the perceptual shifts (sampling rate, length, missingness, channels) across the six benchmarks?

  3. Which components differ per target benchmark: adapters, parsers, evidence-requirement rules, task extensions? Who designed each, and did anyone involved inspect the target questions?

  4. Were the tools implemented by humans or by an LLM? How many were added during source development, and how were the tool families chosen?

  5. Which tools did the ReAct baseline have access to? Was TimeClaw's memory and capability evolution built on TSRBench and frozen, or allowed to update on the targets?

  6. Removing Memory slightly raises average target accuracy (Table 2). In what sense is Memory reusable across benchmarks?

  7. Several ChatTime accuracies in Tables 1, 2, and 10 (e.g., 96.10, 74.13) are not multiples of 1/412. Which question set were these computed on? How many runs per method were excluded as incomplete under Eq. (8)?

4 comments

W1. Establishing the connection between distribution shifts, decoupling, and the proposed components.

(1) Motivation and component mapping.

Figure 1(a) illustrates the cross-dataset transfer gap. Our discussion of perceptual and contextual shifts motivates separating reusable numerical measurements from question-dependent interpretation. The revised experiments examine this design through TPS and evidence-loop ablations and comparisons with ReAct and TimeClaw using shared numerical tools. The components address these challenges as follows:

  • Perceiver, tools, and TPS accommodate input heterogeneity: tools compute measurements from the original numerical series, and TPS exposes them through common statistical and temporal fields with evidence provenance.
  • The Answerer and task extensions accommodate contextual variation: extensions provide question-specific analyses, and the Answerer interprets the resulting evidence under the current question and output requirements.
  • The evidence loop connects contextual needs to additional measurements: unresolved analytical requirements trigger targeted re-perception.

(2) Evidence in the revised paper.

  • Component-wise TPS analysis (Table 3). We separately ablate TPS structure, statistical and temporal fields, semantic summaries, task-specific extensions, and evidence requirements and feedback. These comparisons examine how evidence organization, numerical measurements, and contextual analyses contribute to answering.
  • Evidence feedback analysis (Table 3). Comparisons with and without the loop assess the contribution of additional evidence acquisition. The results show task-dependent benefits, with improvements on some tasks and smaller or mixed effects elsewhere.
  • Shared-tool comparison (Table 5). With the same Qwen3.8-27B backbone and shared numerical tools, TSHarness achieves higher average accuracy across the six benchmarks than ReAct and TimeClaw. This supports a contribution beyond numerical-tool availability alone.

Together, these analyses connect the proposed components to their roles in evidence acquisition and contextual answering, while distinguishing the design motivation from the empirical evidence.

W2. (1) “The claim of ‘remarkable’ gains is not supported by the evidence.” (2) Results come from a single run at temperature 0. The authors should report results over multiple runs

(1) Higher average accuracy.

We revised the contribution statement in the Introduction from “It remarkably outperforms same-backbone baselines without any target-side adaptation.” to “It outperforms same-backbone baselines in average target accuracy without target-side training or answer feedback.”

This makes the claim explicitly about average transfer performance. The expanded Table 1 includes an Avg. column, and added Figure 3 compares CoT, ReAct, TimeClaw, and TSHarness across 26 tasks in 6 datasets. These comparisons reveal task-specific strengths across the evaluated methods: relative rankings vary across capabilities and datasets, and gains can be large, small, or negative.

Section 4.2 explicitly acknowledges this: “The capability profiles show that aggregate transfer gains coexist with task-specific strengths and weaknesses.” We also discuss TSHarness outperforming TimeClaw on TSRBench reasoning while trailing it on CaTSBench retrieval.

Across the five target benchmarks:

  • Qwen3.8-27B: TSHarness achieves 74.95% average accuracy, exceeding TimeClaw, the strongest baseline by average accuracy, at 66.79%.
  • GPT-5.6-sol: TSHarness achieves 79.66%, compared with 76.51% for direct answering.

The supported conclusion is therefore higher average accuracy across heterogeneous targets, with explicitly acknowledged differences across tasks.

(2) Multiple runs.

Reporting main results without run-level standard deviations or confidence intervals also occurs in prior work, including TimeClaw, ChatTime, Time-MQA, and TimeOmni-1. Nevertheless, we agree that the repeated evaluation strengthens the reliability of the comparison.

Section 4.1 of the revised manuscript specifies three-run means for the reported TSHarness and TSHarness* configurations. Table 1 presents the updated averages. We also update the relevant experimental descriptions and results in the appendix to keep them consistent with the revised main text. GPU availability and the rebuttal time limit restricted repeated evaluation of every baseline and ablation. Complete multi-run coverage is planned for a subsequent manuscript update. Small dataset-level margins are not presented as statistically significant improvements.

W3. Coverage and reproducibility of the baseline comparisons.

TSHarness is an agentic framework, so our primary comparisons focus on reasoning and tool-use workflows: CoT as a reasoning baseline, ReAct as a general tool-using agent, and TimeClaw as a closely related time-series agentic method. Additional matched-tool comparisons with ReAct and TimeClaw assess workflow contributions using the same backbone and shared numerical tool library.

The other named methods differ in scope and implementation availability:

  • AION evaluates complete time-series project workflows, including model development and validation. KairosAgent focuses on forecasting and future-morphology reasoning. Applying either to our per-question TSQA setting requires changes to its task interface and evaluation protocol.
  • For TS-Agent, we could not locate an official implementation of the cited paper. An independent reproduction introduces implementation choices that affect comparison fidelity.
  • ChatTS and TimeOmni-1 are relevant specialized models. Our evaluation prioritizes agentic comparisons under the same backbone, with ChatTime and Time-MQA providing representative trained-model comparisons. The comparison is therefore representative rather than exhaustive.

W4 & Q4. Tool-library construction and the role of human engineering.

Tool implementation, tool configuration, and activation-rule development are distinct parts of resource construction.

The base tool library is specified once by humans, covering standard statistical and temporal properties, including descriptive statistics, value distributions, trend, seasonality, volatility, anomalies, change points, segmentation, and cross-series relationships.

During source construction, additional reusable tools were generated using GPT-5.6-sol, consistent with the general approach to LLM-based tool creation explored in LATM [1] and CRAFT [2]. Their implementations were executed, checked on source-development examples, and integrated through the shared tool interface. This stage five specialized tool candidates and introduced two additional diagnostic tools: candidate-trajectory comparison and event-impact interpretation, motivated by analytical needs identified from source execution traces and answer feedback.

The resulting library is frozen and reused across target benchmarks without target-specific tool development. We clarify the construction procedure and tool inventory in Appendices B.1 and B.2.

Reference

[1] Cai et al. Large Language Models as Tool Makers. ICLR 2024.

[2] Yuan et al. CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets. ICLR 2024.

W5. End-to-end efficiency and the contribution of additional test-time compute.

We added total input and output tokens per question to the revised evaluation, alongside accuracy.

Main comparison (Table 1): averages across five target benchmarks.

MethodAvg. accuracy (%) ↑Tokens per question (k) ↓
ReAct64.6617.90
TimeClaw66.7928.75
TSHarness74.9515.82

The revised Main Results section explicitly discusses this accuracy–cost comparison: TSHarness achieves higher average accuracy with 11.6% and 45.0% fewer tokens than ReAct and TimeClaw, respectively. This provides evidence about LLM inference consumption beyond the original tool-execution counts.

Shared-tool comparison (Table 5): averages across six datasets, using 128 sampled questions per dataset and shared numerical tools.

MethodAvg. accuracy (%) ↑Tokens per question (k) ↓
ReAct65.4922.73
TimeClaw52.7338.10
TSHarness72.2717.63

With the same backbone and shared numerical tools, TSHarness achieves higher average accuracy while consuming fewer tokens than ReAct and TimeClaw. These results demonstrate better end-to-end performance and token efficiency in this comparison.

Together, these experiments support a better accuracy–token-cost trade-off relative to the evaluated agentic baselines. Direct zero-shot answering remains cheaper, and token consumption does not by itself establish lower latency or monetary cost.

Q1. Does the Perceiver/Answerer split help when a single agent has identical tools and budget?

(1) Shared-tool comparison with single-agent ReAct (Table 5).

We added a comparison using the same Qwen3.8-27B backbone and shared numerical tools, evaluated on 128 randomly sampled questions per dataset across six datasets.

MethodAvg. accuracy (%) ↑Tokens per question (k) ↓
ReAct65.4922.73
TimeClaw52.7338.10
TSHarness72.2717.63

With the same backbone and shared numerical tools, TSHarness achieves higher average accuracy while consuming fewer tokens than ReAct and TimeClaw. This supports a contribution from pipeline organization beyond numerical-tool access alone.

(2) Evidence organization (Table 3).

We additionally examine evidence organization, individual evidence fields, and feedback. All other listed TPS components are retained unless explicitly removed below.

CaseConfigurationEvidence loopTSRBench: QwenTSRBench: GPTTime-MQA: QwenTime-MQA: GPTChatTime: QwenChatTime: GPT
A1Without TPS structure✗59.3871.8848.4453.1296.8497.33
A2Structured TPS with all evidence fields✗58.5971.8851.5657.0396.6097.57
B1Without statistics✗37.5044.5347.6646.8896.8495.15
B2Without temporal features✗38.2846.0954.6951.5642.4842.48
B3Without textual summaries✗58.5971.0951.5653.1296.3697.57
B4Without task extensions✗35.9452.3449.2256.2597.3388.11
C1Without evidence requirements✗58.5971.8852.3456.2596.3697.57
C2Without evidence requirements✓57.0372.6653.9156.2597.0998.06
FullFull TPS and feedback✓57.8172.6654.6957.0396.8498.30

The revised TPS ablation discussion distinguishes three findings:

  • Evidence organization contributes beyond the presence of evidence fields.

  • Different tasks depend on different evidence.

  • Feedback provides selective benefits.

(3) Dependence on numerical tools (Tables 1–2).

The reviewer correctly observes that removing tools lowers average target accuracy below direct zero-shot answering. The averages across the five targets are:

ConfigurationAvg. target accuracy (%)
Direct zero-shot65.44
TSHarness without tools63.06
Full TSHarness74.95

As discussed in the revised Tools and Memory analysis, numerical tools are a key contributor. Removing them tests the workflow’s dependence on numerical evidence.

The added shared-tool comparison addresses that resource-access concern, while the TPS ablations examine evidence organization and acquisition. Together, these results support the complete tool-grounded workflow.

Q2. Can failures and benchmark shifts be quantified separately?

(1) Separating missing-evidence failures from downstream answering.

The added Required evidence removal and recovery analysis (Figure 5) removes required TPS evidence on ChatTime while keeping the readout fixed:

Evidence conditionQwen3.8-27B accuracy (%)GPT-5.6-sol accuracy (%)
Full evidence90.2996.36
Required evidence removed36.8941.02
After targeted evidence recovery90.2996.36

With the evidence gaps explicitly specified, targeted re-perception restores both scores to the full-evidence level. This isolates the contribution of missing numerical evidence. Figure 4 additionally tracks corrections and regressions after evidence integration.

(2) Quantifying input differences across the six benchmarks.

We analyzed the numerical inputs in the current evaluation benchmark:

BenchmarkLength: median [P25, P75]Length rangeSeries per question: median / maxChannels per episode: max
TSRBench128 [35, 1000]24–10244 / 1212
Time-MQA24 [24, 24]3–241 / 51
ChatTime128 [64, 512]64–5121 / 11
CaTSBench24 [7, 74]3–1502 / 41
TSAQA55 [35, 111]6–3431 / 51
TimeOmni-162 [60, 62]14–7441 / 21

The subsets differ in sequence length and input dimensionality. Length statistics summarize available scalar sequences, and series counts are computed per question with numerical input. Independent candidate series are distinguished from aligned multivariate channels. These statistics characterize the evaluated subsets. We add this table and its definitionto Appendix A.3.

The benchmarks also differ in the task capabilities they assess. The added Figure 3 compares CoT, ReAct, TimeClaw, and TSHarness across 26 tasks from the six benchmarks, showing that the methods have task-specific strengths and weaknesses.

Q3. Which components differ per target benchmark? Who designed each, and did anyone involved inspect the target questions?

(1) How the architecture accommodates different target benchmarks.

  • Adapters and parsers translate benchmark-specific inputs and answer formats into a shared interface.
  • Evidence-requirement rules map the question and task description to the required evidence types.
  • Task extensions and the Answerer adapt the analysis and interpretation to the current question, while retaining the shared TPS structure and registered operations.

(2) Design responsibility and development information.

  • The adapters, parsers, fixed mappings, and extension interface were implemented during human-directed development. The base LLM in the pipeline generates question-specific analysis requests at inference.
  • Target-specific adaptation is limited to implementing adapters and parsers for the published input and answer formats. The source-developed tools and Memory are reused, and all implementations remain fixed during target evaluation.

The added Appendix A.2 clarifies this interface adaptation and the frozen evaluation protocol.

Q5. What tools did ReAct receive, and how was TimeClaw’s evolution controlled?

(1) ReAct’s tool library.

ReAct uses an 11-tool library: channel listing, series overview, channel statistics, value retrieval, autocorrelation, periodicity detection, peak detection, ARIMA forecasting, portfolio VaR, Sharpe ratio, and CAPM regression.

(2) TimeClaw’s source construction and frozen evaluation.

We use TimeClaw’s released implementation. Its episodic Memory is constructed from trajectories generated on TSRBench construction examples, excluding the held-out test split. Target evaluation uses the frozen Memory bank and a fixed tool library, with fingerprint-based retrieval. No target-side Memory updates, tool modifications, or feedback-driven capability evolution are performed. Target reference answers are used only for scoring. This follows the same source-only development and frozen-transfer setting as TSHarness.

(3) Additional comparison with the same tool library.

We give ReAct, TimeClaw, and TSHarness the same numerical-tool library and Qwen3.8-27B backbone. All methods answered the same 128 randomly sampled questions per dataset (Table 5).

MethodAvg. accuracy (%)Tokens/question (k)TSRBenchTime-MQAChatTimeCaTSBenchTSAQATimeOmni
ReAct65.4922.7347.6660.9493.7587.5054.6948.44
TimeClaw52.7338.1039.0653.9181.2588.2832.0321.88
TSHarness72.2717.6360.9465.6398.4498.4453.9156.25

These results show that TSHarness’s average advantage persists under shared tool access.

Q6. What does cross-benchmark reusability of Memory mean when removing it improves average accuracy?

Memory stores procedural guidance developed from TSRBench construction examples, including applicability conditions and recommended analyses. It helps the Perceiver select appropriate tools and analytical procedures. Table 2 shows that removing Memory lowers TSRBench accuracy, supporting its usefulness on the source benchmark.

Reusability refers to deploying this source-developed guidance across benchmarks without target-specific updates. Its benefit depends on whether the guidance matches the target task, so reuse does not guarantee improvement on every benchmark.

We acknowledge this limitation in Conclusion and Limitations: “Procedural memory mainly benefits in-domain performance, while its source-derived priors provide limited support for zero-shot transfer.” Improving Memory’s transferability is identified as an important direction.

Q7. Clarifying evaluation, accuracy computation.

We acknowledge this mistake for agentic methods tested on ChatTime (e.g., 96.10% and 74.13%), which were computed on wrong 4000 questions evaluation subset. We have updated the ChatTime evaluation to use the same fixed 412-question subset across the methods in the main agentic comparison and updated the corresponding results. Besides, We conducted three independent runs for TSHarness and TSHarness* and report their mean accuracies, consistent with the repeated-run evaluation described in W2.

Incomplete executions and scoring. Each main-table agentic run is evaluated on 412 questions. Accuracy is the number of correct answers divided by 412. Missing, invalid, or unparseable answers count as incorrect, and no questions are excluded from the denominator.

We have updated Appendix A.4 (Eq. 8) to clarify the denominator and how these cases are scored.

Reviewer_7w65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

One-line Summary.

This paper introduces TSHarness, a training-free agentic framework for zero-shot time-series question answering (TSQA) across different datasets. It is supported by an intermediate representation called the Time-Series Perception State (TPS).

 

Limitation of Prior Work.

The authors explain two reuse failures in existing TSQA methods:

  • Parameter reuse failure: fine-tuned models overfit to source-dataset correlations.
  • Analytical reuse failure: stored procedures and memories in agent-based methods do not generalize across benchmarks.

 

Core Idea: Decoupling Perception from Contextual Reasoning.

  • Perceiver (tool orchestration): plans calls to numerical tools, guided by a general Memory built on a specific TSQA benchmark (TSRBench, which is not made by the authors).
  • TPS (intermediate representation): a structured record of useful information such as statistical and temporal fields, provenance links, and required evidence types.
  • Answerer: reasons over the TPS, inspects evidence gaps, and produces answers.

 

Effect.

TSHarness is developed on TSRBench only and deployed frozen on Time-MQA, ChatTime, CaTSBench, TSAQA, and TimeOmni-1. With Qwen3.8-27B it outperforms zero-shot answering on all five targets and exceeds CoT, ReAct, and TimeClaw on four of them.

Critical strengths

(1) An interesting and meaningful problem direction

Most TSQA work stays within in-distribution evaluation. Asking whether a system built on one benchmark transfers to others is a practically important question. The attempt to include five targets that differ in input form (single or multi-channel, length, patches) and answer form (multiple choice, true/false, ordered labels) makes the evaluation design comprehensive.

 

(2) Explicit focus on provenance

The TPS keeps source tools and evidence identifiers for every field, so one can in principle trace which measurements the Answerer relied on. As a researcher working on provenance, I find this a solid strength for a reliable agentic system.

 

(3) Clear figures

Figures 1 and 2 make the motivation and the workflow easy to follow. They help a lot in getting an overview of the framework.

Critical weaknesses

(1) Missing comparison between TPS and a single agent given the same tools

The paper's core claim is that the performance gain comes mainly from decoupling perception from reasoning (i.e., the two-step process via TPS), but the experiments do not isolate this from the effect of the tool library.

  • In Table 1, the paper does not state which tools ReAct and TimeClaw received, so the gap in Table 1 may reflect "tools developed on TSRBench vs. generic tools" rather than "decoupled architecture vs. single agent".
  • The findings in Table 2 add to this concern, since most of the performance gain comes from the tools (rather than from Memory).

To support the paper's core claim, the authors should report a comparison in which ReAct and TimeClaw are given the same tool library as TSHarness.

 

(2) Insufficient baselines across backbones

The main experimental results (Table 1 and Table 2) rely on only one backbone, Qwen3.8-27B, which weakens the experimental support. Also, for a fairer comparison, I encourage the authors to run TSHarness, ReAct, and TimeClaw with Qwen3-8B so that they can be compared against the base models of the specialized baselines (Qwen2.5-7B / Llama3-8B / Mistral-7B).

 

(3) Unclear model-size regime in which TSHarness works

A scaling analysis over backbone sizes (e.g., 4B, 8B, 14B, and 27B) seems essential to guide readers on the minimum model size required to run TSHarness. This is an important configuration for TSHarness, since its main strategy is zero-shot transfer.

 

(4) Lack of ablation on the TPS components and the evidence loop

While TPS is proposed as a core contribution, its internal design is not fully ablated. Table 2 only removes Tool and Memory, and no experiment removes S_j, F_j, h_j, the extensions, or Q individually.

 

My current rating is driven primarily by (1), (2), and (3).

Questions for authors

Q1. Why are CoT and ReAct below zero-shot in Table 1?

In Table 1, CoT on TSRBench scores 37.14, which is 6.5 points below zero-shot (43.69), and ReAct is also below zero-shot on TSAQA and TimeOmni-1. Since the paper states that unparseable or format-violating responses are counted as incorrect, I wonder whether this low performance is caused by the token limit applied to the baselines, which can truncate the long chain of thought of a reasoning model before the answer label.

 

Q2. Larger loss on TSAQA in Figure 3(a)

Figure 3(a) shows that TSAQA actually has a negative net change after evidence integration. Could the authors clarify what kinds of errors caused the loss on TSAQA and why this happens?

 

Q3. Stage-level error analysis using the preserved provenance

One of the strengths I pointed out, the preserved provenance, is shown only in principle and not in an actual analysis. I would really appreciate such an analysis, as explainability matters in domains such as medicine where time-series analysis (TSQA) is common.

2 comments

W1. Separating architectural gains from tool-library gains.

In the new shared-tool experiment (Table 5), ReAct, TimeClaw, and TSHarness use Qwen3.8-27B, the same frozen numerical and question-level tools, and the same 128 questions per benchmark. TSHarness runs without the learned Tool Selector. Each method retains its own workflow, and we report actual token consumption.

MethodAvg.Tokens (k)TSRBenchTime-MQAChatTimeCaTSBenchTSAQATimeOmni
ReAct65.4922.7347.6660.9493.7587.5054.6948.44
TimeClaw52.7338.1039.0653.9181.2588.2832.0321.88
TSHarness72.2717.6360.9465.6398.4498.4453.9156.25

TSHarness improves average accuracy over ReAct and TimeClaw while consuming fewer tokens. These results show that the average advantage persists under the same tools.

W2. Matched comparisons across backbones, including Qwen3-8B.

We add a complete Qwen3-8B evaluation in Table 1, bringing the inference backbone closer in size to those used by the specialized baselines.

The evaluation includes Direct, CoT, ReAct, TimeClaw, and TSHarness across all six benchmarks. For each method, only the backbone is replaced, while its main-table configuration is retained.

MethodAvg.Tokens (k)TSRBenchTime-MQAChatTimeCaTSBenchTSAQATimeOmni
Direct50.921.1135.4461.6535.6878.8841.0237.38
CoT54.561.1938.8362.3842.7279.8548.7939.08
ReAct34.2212.1734.9540.7833.0115.0540.5341.75
TimeClaw38.1127.8236.4143.4547.5720.8737.6241.02
TSHarness61.8013.5250.7356.3179.6181.5550.9740.53

At 8B, TSHarness achieves 61.80% average target accuracy, exceeding Direct and the strongest same-backbone baseline CoT.

W3. Model-size dependence and the smallest effective backbone.

We extend the evaluation to Qwen3-4B and Qwen3-8B, alongside the original Qwen3.8-27B results. TSHarness provides accuracy gains with a 4B backbone, the smallest model evaluated.

At each size, we compare Direct, CoT, ReAct, TimeClaw, and TSHarness on the same fixed evaluation subsets. Within each method, we replace the backbone while retaining its main-table configuration.

Table 1 reports the following average accuracy across the five target benchmarks:

BackboneTSHarnessDirectCoTReActTimeClawTSHarness Tokens (k)
Qwen3-4B60.1049.2248.5038.9839.7613.20
Qwen3-8B61.8050.9254.5634.2238.1113.52
Qwen3.8-27B74.9565.4466.7564.6666.7915.82

At 4B, TSHarness improves over direct answering by 10.88 percentage points, with average accuracy only 1.70 points below 8B.

Detailed comparison with a 4B backbone:

MethodAvg.Tokens (k)TSRBenchTime-MQAChatTimeCaTSBenchTSAQATimeOmni
Direct49.221.1132.7754.8533.5077.4339.8140.53
CoT48.501.1836.1759.2234.9560.4445.3942.48
ReAct38.9811.9430.3443.2034.2243.9333.7439.81
TimeClaw39.7629.4535.4442.4842.2338.3535.9239.81
TSHarness60.1013.2050.0054.6178.8875.0046.6045.39

With the same 4B backbone, TSHarness exceeds ReAct and TimeClaw in average target accuracy. It uses 13.20k tokens per question, compared with 11.94k for ReAct and 29.45k for TimeClaw. These results show that the workflow remains useful at a scale where the other evaluated agentic methods struggle.

W4. Ablating TPS components and the evidence-request loop.

(1) TPS structure and component ablations.

The added Table 3 compares flattened and structured records with the same initial evidence and two-stage workflow (A1 vs. A2). Starting from A2, B1–B4 remove statistics, temporal features, textual summaries, or the initial extension block, respectively. The extension block includes the preliminary hypothesis. A2, Full, C1, and C2 cross evidence requirements QQ with loop on/off. Without QQ, the Answerer can still request evidence when the loop is enabled.

Accuracy (%) is shown as Qwen3.8-27B / GPT-5.6-sol.

CaseConfigurationChatTimeTime-MQATSRBench
A1Flattened records, loop off96.84 / 97.3348.44 / 53.1259.38 / 71.88
A2Structured TPS, loop off96.60 / 97.5751.56 / 57.0358.59 / 71.88
B1Remove statistics96.84 / 95.1547.66 / 46.8837.50 / 44.53
B2Remove temporal features42.48 / 42.4854.69 / 51.5638.28 / 46.09
B3Remove summaries96.36 / 97.5751.56 / 53.1258.59 / 71.09
B4Remove initial extensions97.33 / 88.1149.22 / 56.2535.94 / 52.34
C1Remove QQ, loop off96.36 / 97.5752.34 / 56.2558.59 / 71.88
C2Remove QQ, loop on97.09 / 98.0653.91 / 56.2557.03 / 72.66
FullComplete TPS, loop on96.84 / 98.3054.69 / 57.0357.81 / 72.66

Structuring the evidence improves Time-MQA by 3.12 and 3.91 points for Qwen and GPT, respectively. Removing temporal features sharply reduces ChatTime accuracy, while removing statistics or initial extensions substantially lowers TSRBench accuracy. With QQ present, enabling the loop improves Qwen on Time-MQA by 3.13 points, with mixed effects elsewhere.

Q1. Do truncation and answer-format failures explain weaker CoT/ReAct results?

CoT, ReAct, TimeClaw, and TSHarness share the 768-output-token limit per LLM call.

CoT: truncation does not explain the TSRBench gap.

On TSRBench, 0.49% of Qwen3.8-27B CoT responses hit the cap and lacked a parseable answer. Counting all of them as correct would raise CoT from 37.14% to 37.62%, below Direct at 43.69%. CoT improves over Direct on TSRBench with both 4B and 8B backbones (Table 1).

ReAct: task- and backbone-dependent performance.

  • Across tasks: With Qwen3.8-27B, ReAct improves over direct answering on Time-MQA and ChatTime, while underperforming on TSAQA and TimeOmni (Table 1).
  • Across backbones: On TimeOmni, ReAct underperforms direct answering with Qwen3.8-27B, but outperforms it with Qwen3-8B.
  • With shared tools: ReAct outperforms TimeClaw, achieving 65.49% vs. 52.73% average accuracy across six datasets (Table 5).

Q2. Understanding the negative answer revision on TSAQA.

(1) Case analysis.

We analyzed two TSAQA regressions involving an empty anomaly or change-point detector output and partial verification of a candidate Fourier transform. In both cases, the Answerer treated limited measurements as decisive and overrode a correct initial judgment. Additional evidence can therefore hurt when the Answerer draws a stronger conclusion than the measurements support.

(2) Added experiments and broader view analysis.

The added capability-wise comparison across 26 tasks in 6 datasets (Figure 3) provides a broader view of task-dependent performance. The added TPS and evidence-loop ablations in Table 3 likewise show that additional evidence processing has mixed effects across tasks.

Our main evaluation emphasizes average improvement across heterogeneous benchmarks, while retaining these dataset-specific regressions.

Q3. Stage-level analysis using provenance.

We add paired case studies tracing tool measurements → TPS records → cited answer reasoning, covering both successful and failed revisions. TPS preserves the measurements, tool versions, and evidence IDs used by the Answerer.

The added Appendix D.5 (Fig. 9) traces tool measurements through TPS records to the Answerer’s cited evidence in two cases.

  • Successful revision. The tools recorded a rise at indices 73–88 followed by a drop at 110–125. The Answerer cited the corresponding TPS records and corrected down-up to up-down.
  • Failed revision. The anomaly tool recorded a spike and nearby dips in an ECG. The Answerer cited those measurements as evidence of ventricular premature contractions and changed a correct FALSE to TRUE. The trace locates the error in how the Answerer interpreted the detector output.
Reviewer_TZefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the overfitting and generalization issues of specific time-series QA models, this paper presents an agentic framework to decouple the workflow of TSQA into numerical perception and contextual reasoning. Instead of outcome feedback, the proposed harness adopted evidence supplement to trigger iterative perception-reasoning procedure, so as to providing a generalizable, well-performance foundation for zero-shot TSQA.

Critical strengths
  1. The author decomposed a monolithic task TSQA into numerical perception and contextual reasoning, them conquer them vis time-series perception state (TPS).

  2. To eliminate the need of outcome feedback, the proposed TSHarness launched a inner optimization by supplying the evidence of TPS iteratively like the procedure of deepsearch.

Critical weaknesses
  1. In the harness loop part, the gap inspection (whether the required evidence is present) is the engine of this loop, but there is no accuracy validation of it. W/O such validation, it is hard to analyze the bottleneck of Qwen3.8-27B and GPT-5.6-sol.

  2. The author should provide details about the training configure of the tool selector since the tool selector is the basic part of TPS. Meanwhile, the tool selector was trained via PPO, the reward model was not mentioned. [hints, I did't find them in the appendix partition.]

  3. Lack of depth analysis in sub-section 4.2 and 4.3. For examples:

  • why CoT and ReAct underperform zero-shot as shown in Table 1.
  • It is abnormal in Table 1, the in-domain training results of different generations (qwen2.7-7B, llama3-8B, and Mistral-7B) is similar on Time-MQA, ChartTime and TimeOmni.
  • Why the result of w/o memory is best on Time-MQA and ChatTime ?
Questions for authors
  1. The learned tool selector was embedded in TSHarness ?

  2. Was the tool selector based on Qwen3.8-27B ?

  3. I'm confused by the analyses in sub-section 4.5, Can you provide concise version about the accuracy and efficiency regimes ?

2 comments

W1. Accuracy validation of gap inspection and analysis of the model bottlenecks.

(1) An additional verification.

We evaluate gap inspection on ChatTime, where each question type maps to one of four evidence categories. We compare:

  • ALL: Retain all four tool-evidence categories.
  • w/o Required Evidence: Remove the evidence required by the question.
  • w/o Unrelated Evidence: Remove an unrelated category evidence.

The runtime recomputes evidence coverage after each intervention and supplies the missing-evidence list to the Answerer. We exclude some information and procedural memory to prevent duplicate representations from revealing the removed evidence.

a. Evidence-request rates.

TPS conditionQwen3.8-27BGPT-5.6-sol
ALL0%0%
w/o Required Evidence100%100%
w/o unrelated Evidence0%0%

b. Answer accuracy. The table reports accuracy before evidence recovery.

BackboneALLw/o Required Evidencew/o unrelated Evidence
Qwen3.8-27B90.29%36.89%92.23%
GPT-5.6-sol96.36%41.02%94.90%

Both backbones recovered the removed evidence in every required-category case, restoring accuracy to the ALL level (Fig. 5). This validates request and recovery for known evidence requirements.

(2) Model bottlenecks.

With all four evidence categories available, Qwen still makes 40 errors, compared with 15 for GPT. Qwen’s errors concentrate on seasonality (19) and outliers (17), accounting for 36 of its 40 errors; GPT makes 4 and 6 errors in these categories, respectively.

Since both backbones detect and repair every introduced gap, the remaining failures occur after evidence acquisition. This points to the adequacy of tool-generated evidence and its interpretation as the next bottlenecks to examine. These results distinguish successful gap inspection from successful answering, without assuming that every remaining error is a reasoning failure.

W2 & Q1–Q2. Tool selector: integration, architecture, and training.

TSHarness* uses the learned selector to refine the Perceiver’s initial tool plan. Standard TSHarness executes the plan without this refinement. As described in Sec. 3.2, the selector optionally refines the Perceiver’s initial tool plan before execution.

The selector is a 396→64→1 tanh MLP (Appendix C.1). Qwen3.8-27B generates rollouts with frozen weights, and PPO updates only the MLP.

The reward is computed directly from answer correctness and base-tool cost (Eq. 7), without a learned reward model:

Rtool=1[y^=y]−λtoolNbaseR_{\mathrm{tool}} = \mathbf{1}[\hat{y}=y] - \lambda_{\mathrm{tool}}N_{\mathrm{base}}

SettingValue
Source training / development questions96 / 48
Training passes1
Rollouts per question / questions per batch4 / 4
Optimizer / learning rateAdam / 0.003
PPO clip / update epochs per batch0.2 / 4
Prior KL weight / entropy weight0.02 / 0.001

Appendix C and Table 11 give the full configuration. Checkpoint selection uses source-development accuracy, with mean reward breaking ties. The selected policy is frozen for target evaluation.

Tool-cost ablation on TSRBench (Table 4, mean ± SD over three training seeds)

λController Acc. (%)Selector Acc. (%)Controller calls/questionSelector calls/question
058.07 ± 0.4558.33 ± 0.458.930 ± 0.0008.930 ± 0.000
0.0157.81 ± 0.7858.07 ± 2.968.919 ± 0.0051.878 ± 0.005

Without the penalty, the selector executes as many tools as the Controller. With λ = 0.01, it performs 78.9% fewer tool executions with similar mean accuracy, supporting the role of the explicit cost penalty in learning selective tool use.

W3. Depth analysis in sub-section 4.2 and 4.3.

We supplement the results in Sections 4.2 and 4.3 with additional experiments and in-depth analyses.

a. Why do CoT and ReAct sometimes underperform zero-shot?

The benefits of reasoning and tool use are task-specific in TSQA. Both CoT and ReAct improve over direct answering on Time-MQA and ChatTime (Table 1). CoT can propagate an initial numerical misread, and ReAct depends on choosing relevant measurements and interpreting them correctly. For example, CoT and ReAct show different strengths even within TSRBench:

TSRBench QA typeCoT Acc. (%)ReAct Acc. (%)
Reasoning32.1628.07
Prediction38.8945.37

We add a comprehensive radar plot (Figure 3) comparing CoT, ReAct, TimeClaw, and TSHarness across 26 task/domain categories from six benchmarks. This detailed breakdown reveals task-specific strengths and weaknesses.

b. Why do the specialized models obtain similar accuracies?

Table 1 includes task-specialized checkpoints with two training domains:

  • Time-MQA-trained models: Qwen2.5-7B, Llama3-8B, and Mistral-7B. Time-MQA dataset is their in-domain evaluation.
  • ChatTime-trained model: The LLaMA-2-7B-based ChatTime checkpoint. ChatTime dataset is its in-domain evaluation.

The three Time-MQA-trained checkpoints share task-specific training and a comparable scale of 7–8B, which can yield close aggregate scores.

However, their performance differs across capabilities. We further analyze their original predictions on the same TimeOmni questions:

CapabilityQwen2.5-7BLlama3-8BMistral-7B
Causality discovery31.82%37.50%30.68%
Scenario understanding35.39%36.52%36.52%
Decision making22.41%10.34%18.97%
Overall32.04%33.25%31.55%

These differences partially offset in the overall score, showing that similar average accuracy can mask different strengths across capabilities.

c. Why does removing memory improve accuracy?

Memory provides procedural guidance developed on TSRBench and kept fixed during zero-shot transfer. As discussed in Section 4.3, source-developed guidance transfers unevenly across tasks.

Guidance that matches a question’s analytical requirements can help select appropriate tools. However, guidance that is poorly matched to a target question may lead to omitting useful evidence, even when the necessary tools are available. Removing Memory can avoid this interference on such questions.

We expand this discussion in Section 4.3 and explicitly acknowledge it in Conclusion and Limitations: procedural memory mainly benefits in-domain performance, while its source-derived priors provide limited support for zero-shot transfer. Improving the transferability of memory is an important direction for future work.

Q3. Clarifying the accuracy and efficiency regimes.

(1) Accuracy–efficiency trade-off settings.

Both regimes use the same trained checkpoint. The parameter α\alpha scales the learned logit adjustment. All configurations use a 0.5 inclusion threshold and retain the highest-probability tool if none passes (Appendix C.3).

  • Accuracy-oriented (TSHarness*): Prioritize evidence coverage, using a moderated adjustment and additional retention rules to preserve basic numerical evidence.
  • Efficiency-oriented: Prioritize fewer tool executions, applying the full learned adjustment without additional retention rules.
  • Intermediate configurations: Vary adjustment strength and minimum tool retention to examine the trade-off between these objectives.
Configurationα\alphaMinimum retainedRetain value profile + statistics
Efficiency-oriented1.0NoneNo
Intermediate 10.7NoneNo
Intermediate 20.4NoneNo
Intermediate 30.42No
Accuracy-oriented0.42Yes

(2) How efficiency is measured.

Efficiency measures the reduction in tool executions across all stages and input series, relative to the matched no-selector Controller:

Tool savings = (1 − selector executions / Controller executions) × 100%.

In our paper, Figure 6(a) reports the following operating points. Relative tool executions are normalized to the corresponding Controller (100%).

Operating pointAccuracyRelative tool executionsTool savings
Efficiency-oriented72.54%43.61%56.39%
Intermediate (left)73.62%57.43%42.57%
Intermediate (middle)74.10%74.28%25.72%
Intermediate (right)74.55%85.23%14.77%
Accuracy-oriented74.71%98.24%1.76%

The reported endpoints illustrate the accuracy–efficiency trade-off: accuracy-oriented deployment retains more evidence and achieves 2.17 percentage points higher accuracy, while efficiency-oriented deployment reports substantially greater execution savings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.