acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Conditioned Multi-Modal Retrieval for Time-Series Analysis with In-Context Learning

Abstract

Time-series analysis increasingly combines numerical trajectories with textual context, yet the two modalities provide complementary and unevenly reliable evidence. Existing language-model pipelines improve temporal representation or retrieve demonstrations through lexical, semantic, or temporal similarity, but they usually apply one retrieval rule to every query or require downstream model feedback to learn selection. We introduce Reliability-Conditioned Query Adaptive Similarity for Hybrid retrieval (RC-QASH), a multi-modal In-Context Learning (ICL) approach that treats text/time-series fusion as a conservative query level reliability decision. Using only nested out-of-fold (OOF) records from the demonstration bank, its gate summarizes view-specific class distributions, confidence, disagreement, and ranking overlap, then either retains the Hybrid anchor or shifts the modality balance. On Finance and Weather, the full pipeline achieves the best cross-domain point estimates in the DeepSeek baseline comparison, with equal-domain mean Accuracy of 57.75% and PRIMARY of 49.19%, where PRIMARY averages balanced accuracy and macro-F1. Relative to Time-Series Top-3, the strongest reported comparator on these aggregate metrics, these scores correspond to improvements of 6.20% and 6.52%, respectively. Shared banks, prompt rendering, and fixed decoding settings isolate retrieval changes. Matched DeepSeek and Llama comparisons evaluate the controller on changed prompts. The gate concentrates its changes on queries where static Hybrid retrieval is weaker, and these comparisons show positive mean gains within the selected subsets. These results show that retrieval geometry can support selective multi-modal adaptation without downstream-output supervision.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
time-series analysis, retrieval-augmented learning, reliability-aware learning, in-context learning, multi-modal learning, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q2arOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce RC-QASH, a method across time-series and text for in-context learning. The retrieval similarity is a mixture of time series and text similarity. RC-QASH has a module trained to predict whether the given query leans towards temporal data, text data or a default mixture described by the authors as anchor.

The method is evaluated on the Finance and Weather datasets of MTBench. The achieved results are measured in accuracy and a mean of macro F1 score and balanced accuracy. The DeepSeek variant seems to improve across domains on the measured benchmark.

Critical strengths
  • S1 Comparison of the proposed retriever method across Finance and Weather on MTBench, LLM backbones, and ranking to isolate the performance gains.
  • S2 The authors state the positive mean gains on the changed samples compared to the default/anchor retriever helping assess the gain of the gate.
Critical weaknesses
  • W1 Statistical power of the results: 43 changes compared to Hybrid in Finance are arguably few with limited statistical power to answer whether there is a gain through the proposed method. A statistical analysis should be presented in the main part of the paper. In case of insufficient statistical power, more independent datasets would strengthen the evidence.
  • W2 Choice of text and time-series similarity weights: It is unclear how the authors came up with the αts\alpha_{ts}, αanchor\alpha_{anchor}, and αtext\alpha_{text}. There should be a transparent evaluation of different values, reporting the performance of the gate and whether it leans towards time-series, text similarity, or towards the anchor.
Questions for authors
  • Q1 Can you please provide a power analysis to establish how the results for both Weather and Finance should be interpreted?
  • Q2 Can the authors provide their interpretation of the difference in order of “random” and “vanilla” in figure 2 a) and b)? Why is no additional retrieved context (vanilla) better for Weather but worse for Finance?
  • Q3 What is the performance when trained on Finance and Weather all at once instead of separately?
  • Q4 How does the performance look across different weighting of time-series and text similarity? Out of the changes decided by the gate, how many are towards text and how many towards time-series similarity across different similarity weights?
Reviewer_dNcoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes RC-QASH, a query-adaptive demonstration retriever that uses retrieval reliability features and a lightweight controller trained on demonstration-bank proxy labels to select among three text/time-series mixture weights, without downstream LLM supervision. Experiments on MTBench’s Finance and Weather tasks with DeepSeek and Llama evaluate overall retrieval performance and controller effects, reporting improvements over single-modality baselines but small, mostly statistically inconclusive gains over fixed Hybrid retrieval.

Critical strengths

The controller is trained entirely from the labeled demonstration bank without downstream LLM feedback, avoiding repeated model calls to obtain training supervision. This makes the approach lightweight and potentially useful when downstream evaluations are costly.

Critical weaknesses
  1. Very limited technical novelty. The method combines hand-crafted retrieval features, logistic regression over three preset mixture weights, and a confidence threshold with fallback. These are straightforward design choices, making the overall contribution highly incremental.

  2. Small and statistically inconclusive controller gains. Against fixed Hybrid retrieval, the controller improves DeepSeek accuracy by only 0.18 percentage points on Finance and 1.12 on Weather; PRIMARY gains are just 0.26 and 0.82 points. All corresponding confidence intervals include zero, leaving the core component’s effectiveness unestablished.

  3. The necessity of dynamic routing is not demonstrated. A bank-selected fixed mixture achieves a higher point estimate on Weather, and neither domain shows a significant advantage over this baseline. The results therefore do not establish that query-dependent routing provides sufficient value beyond selecting a suitable fixed weight.

Questions for authors
  1. What technical novelty distinguishes RC-QASH from a standard feature-based gating model?
  2. Given the small gains and confidence intervals crossing zero, what evidence establishes the controller’s effectiveness over fixed Hybrid retrieval?
  3. What evidence demonstrates the necessity of query-dependent routing over a bank-selected fixed mixture?
Reviewer_sn7nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RC-QASH, a query-adaptive demonstration retrieval method for in-context learning on multimodal time-series prediction tasks. The method combines text and time-series similarities using three preset fusion weights. A logistic regression gate uses retrieval statistics, including neighbor-label distributions, confidence, cross-modal disagreement, and ranking overlap, to choose a weight for each query. The gate is trained using nested out-of-fold records from the labeled demonstration bank, with neighbor-label prediction outcomes providing supervision without downstream LLM feedback. A confidence threshold favors retaining the default hybrid retrieval when evidence for adaptation is insufficient. Experiments cover the Finance and Weather domains of MTBench, primarily using DeepSeek-V3-0324, with additional evaluation on Llama-3.3-70B-Instruct. The authors report gains over several retrieval baselines and analyze the controller’s effects relative to static hybrid retrieval.

Critical strengths

The controller can be trained independently of downstream LLM feedback. RC-QASH generates supervision by leveraging out-of-fold neighbor-label predictions derived from the labeled demonstration repository and trains a lightweight logistic regression gate mechanism to calibrate the equilibrium between text and time-series retrieval. This resolves a significant practical limitation in demonstration selection: acquiring model-specific feedback necessitates considerable access to the downstream decoder. The capacity to train the controller utilizing solely the demonstration repository represents a substantial advantage for scenarios where such access proves prohibitively expensive or unavailable, and represents the most compelling justification supporting this methodology.

Critical weaknesses
  1. The empirical evidence fails to convincingly demonstrate the superiority of adaptive routing relative to static fusion. On DeepSeek, the proposed controller achieves only marginal improvements in full-test PRIMARY performance compared to the static Hybrid baseline, with gains of merely 0.26 percentage points on Finance and 0.82 on Weather, and both 95% confidence intervals encompassing zero (Appendix H). Furthermore, the primary comparison table excludes static hybrid retrieval entirely. The bank-selected static fusion comparison presented in Appendix D relies on a single experimental run and produces a higher point estimate for Weather. These findings render the central contribution inadequately substantiated: the reported headline improvements relative to unimodal retrieval do not establish that query-adaptive routing provides a dependable advantage over an appropriately calibrated fixed fusion weight.

  2. The evaluation demonstrates limited evidence regarding general applicability. The experimental scope encompasses only the Finance and Weather domains from MTBench, comprising 375 and 980 external test queries respectively. The primary evaluation employs DeepSeek-V3-0324, whereas cross-backbone validation consists of a single experimental run utilizing Llama-3.3-70B-Instruct. Given the modest performance gains achieved by the controller, this restricted scope impedes the determination of whether improvements generalize across independent benchmarks, alternative model architectures, and replicated decoding iterations. This substantially constrains the evidence supporting the method's broader applicability to multimodal time-series analysis.

Questions for authors
  1. The gains from the gate over static Hybrid are small, and the confidence intervals include zero. What do the authors see as the strongest evidence that the gate is useful? I’d like to understand how much of the improvement comes from combining text and time-series information, and how much comes from adjusting their weights for each query.

  2. Most of the evaluated baselines are basic retrieval strategies. The paper discusses trained demonstration retrievers such as EPR and UDR, but does not evaluate them. Why were these methods left out, and how were the baselines selected? This matters because the current comparisons leave the method’s performance against competitive retrieval approaches unclear.

  3. Finance and Weather both come from MTBench, which leaves the evaluation fairly narrow. I’d like to see results on several other independent datasets, with enough test examples in each to draw reliable conclusions. Given the small improvements reported here, broader coverage would help show whether the method works across different settings or whether its benefits depend heavily on this particular benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.