- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether time-series QA systems actually use the numerical values they are given, rather than only evaluating accuracy. The paper separates three aspects, including task performance, response to changed numerical evidence, and rationale reliability. It evaluates four TSQA systems with several controlled interventions. I think the question is relevant and the observations are useful, but overall the work feels more like a careful diagnostic study than a sufficient contribution.
Critical strengths
S1. Separating task accuracy, evidence-responsive behavior, and rationale reliability is important and useful for evaluation.
S2. The authors provide several observations when analyzing similar accuracy and rationale-answer agreement.
S3. The authors construct the controlled experiments.
Critical weaknesses
W1. The novelty is limited. The main methodology is essentially controlled input and ablation. Reliability and faithfulness are already common in evaluation. So authors just apply these to TSQA. And the COMMON-TSQA is mainly aggregation rather than a new design.
W2. Many conclusions are also not surprising for TS researchers, especially for TSQA models suffer from LLM reliability problems.
W3. The interventions are not cleanly separated. It would be hard to tell exactly what kind of temporal information the model relies on.
W4. Several necessary and deeper analyses only apply to particular systems or subsets. For example, the rationale audit.
W5. Simple statistical time series baselines and non-TS LLMs should be included under the same setup.
W6. The paper also lacks uncertainty or significance analysis as some reported differences are small.
Questions for authors
See Weaknesses.