acceptodds
Withdrawn from ICLR 2027

Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent years, time-series question answering (QA) systems have made significant progress. However, generating a correct answer does not show whether retaining the supplied numerical series improves task performance, nor whether the prediction is sensitive to changes in that input. While some systems provide rationales, answer accuracy also does not show whether their numerical claims are grounded in the supplied series or whether the stated inference is valid. In this work, we focus on evaluating four time-series QA systems: TimeOmni-1, ChatTS, TimeOmni-VL, and Time-MQA. First, for three systems with released evaluation data, we reproduce their reported results and compare the performance of the systems with their backbones. Then, we introduce a benchmark named COMMON-TSQA, which collects public evaluation datasets from existing time-series benchmarks and unifies their sample representation, task definitions, and answer schemas, while evaluating each system through its own interface under common evaluation criteria. The evaluation uses the original condition and six interventions while keeping the question and target fixed. Our analysis shows that aggregate performance alone can obscure how systems use numerical evidence. Similar task-level scores can arise despite substantial changes in individual predictions. Some interventions induce simple fallback behavior rather than preserved task ability. We also evaluate rationales for factual grounding, inference validity, and consistency with the final answer. We find that rationales often contain time-series claims unsupported by the input. Moreover, the rationale audit shows that agreement between a rationale and its final answer can coexist with incorrect numerical descriptions or invalid intermediate inferences.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Question Answering, Time-Series Reasoning, Reasoning Validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qTUBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether time-series QA systems actually use the numerical values they are given, rather than only evaluating accuracy. The paper separates three aspects, including task performance, response to changed numerical evidence, and rationale reliability. It evaluates four TSQA systems with several controlled interventions. I think the question is relevant and the observations are useful, but overall the work feels more like a careful diagnostic study than a sufficient contribution.

Critical strengths

S1. Separating task accuracy, evidence-responsive behavior, and rationale reliability is important and useful for evaluation.

S2. The authors provide several observations when analyzing similar accuracy and rationale-answer agreement.

S3. The authors construct the controlled experiments.

Critical weaknesses

W1. The novelty is limited. The main methodology is essentially controlled input and ablation. Reliability and faithfulness are already common in evaluation. So authors just apply these to TSQA. And the COMMON-TSQA is mainly aggregation rather than a new design.

W2. Many conclusions are also not surprising for TS researchers, especially for TSQA models suffer from LLM reliability problems.

W3. The interventions are not cleanly separated. It would be hard to tell exactly what kind of temporal information the model relies on.

W4. Several necessary and deeper analyses only apply to particular systems or subsets. For example, the rationale audit.

W5. Simple statistical time series baselines and non-TS LLMs should be included under the same setup.

W6. The paper also lacks uncertainty or significance analysis as some reported differences are small.

Questions for authors

See Weaknesses.

Reviewer_6S3pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether final-answer performance in time-series QA reflects effective use of numerical evidence, and whether visible rationales are factually grounded and logically valid. The authors first reproduce the native evaluations of three specialized systems and compare them with their corresponding backbones. They then assemble COMMON-TSQA, comprising 2,600 samples across six tasks, and evaluate four systems under six numerical-evidence interventions. Further analyses distinguish instance-level prediction changes, simple fallback behavior, and cases where an intervention changes the correct answer. For two systems with visible rationales, the authors separately assess factual grounding, inference validity, and conclusion consistency, and validate the LLM judge against a human-annotated subset. The results show that specialization gains do not guarantee reliable evidence use: similar aggregate accuracy can conceal substantial prediction turnover, while agreement between a rationale and its final answer does not establish the reliability of its numerical claims or inferences.

Critical strengths

S1. By combining instance-level prediction changes, single-class fallback analysis, and post-intervention target recomputation, the authors distinguish input sensitivity from appropriate evidence use. In particular, target recomputation demonstrates that scoring against original labels can misclassify correct updates as errors. These analyses avoid inferring evidence use solely from aggregate score changes and provide useful methodological guidance for future evaluations.

S2. Native evaluation reproduction, interventions on a common benchmark, and visible-rationale audits examine specialization gains, evidence responsiveness, and explanation reliability, respectively. Separating factual grounding, inference validity, and conclusion consistency reveals failures that answer correctness or rationale consistency alone would miss. Blinded human validation supports the automated judge, while the authors appropriately avoid equating visible reasoning errors with deficiencies in internal mechanisms.

Critical weaknesses

W1. Table 1 scores all modified inputs against the original labels, although interventions can change the correct answer. For example, flattening two series makes their amplitudes equal, so a correct “equal” response may be penalized against the original target. The authors appropriately address this issue on an exact-oracle cross-series subset, but comparable checks are missing for other affected tasks. Consequently, score drops outside this subset cannot distinguish model errors from correct responses to changed evidence. The paper should clarify which task–intervention combinations preserve, change, or make the target indeterminate.

W2. The factual-grounding prompt and blinded human validation are useful, but the total audit size, task distribution, and complete decision rules for inference validity and conclusion consistency remain unclear. Moreover, strict output-level grounding may penalize longer rationales simply because they contain more claims and therefore more opportunities for error. Without claim-count distributions or length-controlled analysis, the low pass rates are difficult to interpret precisely. Providing the full protocol and separating contradicted from unverifiable claims would improve reproducibility and clarify the severity of the observed failures.

W3. Flat, Zero, and Noise may introduce atypical inputs while removing temporal information, whereas Swap may create inconsistencies between the replacement series and the retained textual context. Thus, behavioral changes may reflect altered evidence, distribution shift, or contextual conflict, rather than a single identifiable limitation in temporal understanding. This does not invalidate the behavioral findings, but limits their interpretation. Natural local edits with controlled answers, together with answer-preserving controls, would help distinguish appropriate evidence use from sensitivity to abnormal or inconsistent inputs.

Questions for authors

Q1. Which task–intervention combinations preserve the correct answer, which change it, and which make it indeterminate? Please clarify the size, selection criteria, and relation-type coverage of the target-recomputation subset, and explain how these limitations affect the interpretation of Table 1.

Q2. Could the authors provide the complete decision rules for inference validity and conclusion consistency, together with the total numbers of audited outputs and claims by system and task? Do the strict grounding results remain consistent after controlling the number of claims? and how frequently are claims contradicted and how frequently are unverifiable?

Q3. Does Swap compatibility screening ensure that replacement series remain consistent with the retained textual context? What evidence helps distinguish responses to changed temporal information from responses to atypical inputs or contextual conflicts?

Q4. What accounts for the difference between Time-MQA’s recomputed-target accuracies in Tables 10 and 15 (48.88% versus 38.27%)? Please clarify, if there is any difference in model revision, input formatting, decoding, parsing, or evaluated samples.

Reviewer_2mALOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the extent to which TimeOmni-1, TimeOmni-VL, Time-MQA, and ChatTS rely on the numerical data provided to them and evaluates the factual validity of their explanations. The analysis is based on a series of interventions. To enable a unified evaluation, the authors introduce Common-TSQA, a benchmark for time-series question answering.

The main contributions of this work are:

Reproduction and comparison: Reproducing the results of selected time-series specialist models and comparing their performance against baseline methods.

Common-TSQA benchmark: Introducing a unified benchmark for evaluating time-series understanding across different systems, comprising 2,600 samples from six task categories.

Intervention-based evaluation: Applying six intervention types—Shuffle, Flat, Noise, Answer-only, Swap, and Zero—to assess the extent to which models depend on numerical data.

Critical strengths

This work introduces a unified evaluation framework that enables systematic comparisons across different models and architectures. The authors standardize the source data formats and then transform them into the representations best suited to each model, including text, images, and specialized encodings. This approach facilitates a more consistent and fair comparison across different systems.

Critical weaknesses
  1. Limited novelty

The novelty of the Common-TSQA benchmark and the overall contributions of this work could be better justified. The study builds heavily on existing models, including TimeOmni-1, TimeOmni-VL, Time-MQA, and ChatTS, as well as previously published datasets. The authors should more clearly articulate how their proposed benchmark and evaluation methodology differ from prior work and what new insights they provide beyond existing approaches.

  1. Evaluation concerns

The evaluation has several limitations and raises some questions about the experimental design:

Limited uncertainty quantification and uneven task coverage: The paper provides limited information about the statistical uncertainty of the reported results, while the coverage of different task categories appears uneven.

Lack of intervention-intensity analysis: The authors do not investigate how varying the intensity of interventions affects model performance. This limits the conclusions that can be drawn about model sensitivity to different levels of perturbation, such as increasing noise levels.

Questions for authors

Statistical uncertainty: Could the authors provide confidence intervals for selected key results? Ideally, these should account for dependencies between examples, for instance, by grouping or clustering examples according to their source time series or dataset.

Factual grounding and scoring: How is a response scored when even a single claim is classified as "not verifiable" during factual grounding? Additionally, how often are responses with the correct multiple-choice answer (A/B/C/D) rejected or penalized during the factual grounding evaluation?

Common factual grounding errors: What are the most frequent types of errors identified during factual grounding? Could the authors provide a quantitative breakdown of these errors, along with representative examples or case studies to illustrate the most common failure modes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.