- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce RC-QASH, a method across time-series and text for in-context learning. The retrieval similarity is a mixture of time series and text similarity. RC-QASH has a module trained to predict whether the given query leans towards temporal data, text data or a default mixture described by the authors as anchor.
The method is evaluated on the Finance and Weather datasets of MTBench. The achieved results are measured in accuracy and a mean of macro F1 score and balanced accuracy. The DeepSeek variant seems to improve across domains on the measured benchmark.
Critical strengths
- S1 Comparison of the proposed retriever method across Finance and Weather on MTBench, LLM backbones, and ranking to isolate the performance gains.
- S2 The authors state the positive mean gains on the changed samples compared to the default/anchor retriever helping assess the gain of the gate.
Critical weaknesses
- W1 Statistical power of the results: 43 changes compared to Hybrid in Finance are arguably few with limited statistical power to answer whether there is a gain through the proposed method. A statistical analysis should be presented in the main part of the paper. In case of insufficient statistical power, more independent datasets would strengthen the evidence.
- W2 Choice of text and time-series similarity weights: It is unclear how the authors came up with the , , and . There should be a transparent evaluation of different values, reporting the performance of the gate and whether it leans towards time-series, text similarity, or towards the anchor.
Questions for authors
- Q1 Can you please provide a power analysis to establish how the results for both Weather and Finance should be interpreted?
- Q2 Can the authors provide their interpretation of the difference in order of “random” and “vanilla” in figure 2 a) and b)? Why is no additional retrieved context (vanilla) better for Weather but worse for Finance?
- Q3 What is the performance when trained on Finance and Weather all at once instead of separately?
- Q4 How does the performance look across different weighting of time-series and text similarity? Out of the changes decided by the gate, how many are towards text and how many towards time-series similarity across different similarity weights?