acceptodds
Under review as a conference paper at ICLR 2027

TimeInteract: Towards Real-Time Interactive Intelligence for Streaming Time Series

Abstract

Real-world time series evolve continuously, with meaningful changes potentially emerging at any moment. However, existing time-series language models (TSLMs) remain inherently static. They either receive complete sequences for offline processing or alternate between streaming input and response generation, which prevents processing of new observations during interaction. We introduce a new regime, **Time-Series Interaction**: a model continuously perceives incoming time-series observations and user intent, autonomously decides when to remain silent or respond, and continues processing new observations during response generation. To realize this, we develop **TimeInteract** with three key designs: a *dual-view streaming TS encoder* that captures local variations and historical dynamics, a *response control mechanism* that learns when to trigger a response, and a *decoupled streaming inference mechanism* that separates control from response generation to avoid blocking subsequent observations. We further formulate a hierarchy of interaction capabilities, progressing from Understanding to Adaptivity. Based on this hierarchy, we construct **StreamTSI-34K**, a large-scale streaming TS interaction dataset with 34,588 episodes and 77,505 responses across synthetic and real-world time series in single- and multi-turn settings. Across all four interaction levels, **TimeInteract** consistently outperforms existing LLMs, VLMs, and TSLMs, with gains of up to 23.92 points on challenging tasks. It also improves response triggering while achieving near-zero stream stall and up to inference speedup.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Time Series Language Model, Multimodal Time Series

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_w9nKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies interaction with a streaming time series, where the model decides on its own when to respond and keeps reading new data while it generates. It makes three contributions.

  • TimeInteract builds on a small instruction-tuned LLM and adds a fast/slow stream encoder, a control head with compact Plan tokens that summarize earlier replies, and a forked KV cache that lets generation run without blocking the stream.
  • A four-level task hierarchy covers answering questions, following standing instructions, warning proactively and adapting to user feedback.
  • STREAMTSI-34K is a dataset built from synthetic and real series, with interactions constructed from templates, rules and LLM generation and then checked by an LLM verifier.

TimeInteract is trained on this dataset and, on its held-out split, scores highest on every task metric and on response-triggering F1 against a range of existing LLMs, vision-language models and time-series LMs. It also completes responses faster than interleaved generation.

Critical strengths
  • I find the problem setup useful. The paper turns streaming interaction, where the model has to decide when to respond while data keeps arriving, into concrete task levels that can be measured.
  • The efficiency results support the decoupled inference design, and I think it is the most solid part of the paper. Forking the KV cache brings the stall during generation close to zero and shortens completion time relative to interleaved generation, especially for longer responses. Time to first token stays about the same.
Critical weaknesses
  • My main concern is that the main comparison can't isolate the method.
    • TimeInteract is fine-tuned on STREAMTSI-34K and tested on its held-out split, and as far as I can tell none of the baselines is trained on this data.
    • Apart from each baseline's input type, I couldn't find a description of how the baselines run in the streaming setting, such as their prompts and how the stream is presented to them.
    • Because of this, I don't think the main tables can separate the benefit of the architecture from the benefit of training on the benchmark's own format. I'd suggest adding a baseline fine-tuned on the same data, ideally the same backbone without the proposed components.
  • I'm also not fully comfortable with the evaluation, which is small and under-described.
    • The paper doesn't say who or what assigns the semantic correctness score.
    • The main tables come from a small test sample, and they report no variance estimates, no repeated runs and no per-task test counts.
    • The test labels come from the same construction pipeline as the training data, which relies on LLM generation and LLM verification, and humans review only the low-confidence cases, so I'd be careful about reading the scores as measures of real interaction quality.
Questions for authors
  • How exactly is each baseline run in the streaming setting? Have you fine-tuned any baseline on STREAMTSI-34K?
  • How is semantic correctness scored? If an LLM judge is used, which model and prompt?
  • How many test items does each task have, and how much do the results vary across training seeds?
8 comments

We thank the reviewer w9nK for the careful assessment and constructive suggestions. We provide detailed point-by-point responses below and hope that our clarifications and additional evidence will fully address your concerns.

W1 My main concern is that the main comparison can't isolate the method.

To better isolate the contributions of our proposed architecture, we add two groups of controlled architectural ablations. All variants, including TimeInteract, are trained on the same 10% subset of STREAMTSI-34K with an identical training budget to ensure fair comparisons.

a) Encoder Ablations. We consider two alternatives to the proposed dual-stream Mamba encoder:

(1) Single-stream Mamba. We replace the dual-stream encoder with a single-stream Mamba encoder to evaluate the benefit of dual-stream modeling.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Single-stream Mamba69.935.671.333.4
Ours70.841.276.136.7

Results. The dual-stream encoder outperforms single-stream Mamba across all four metrics.

(2) Alternative Encoders. We replace Mamba with CNN and Transformer encoders to examine the impact of the encoder architecture.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
CNN66.429.867.627.1
Transformer70.234.371.932.8
Ours70.841.276.136.7

Results. TimeInteract achieves better performance than both CNN and Transformer alternatives.

b) Response Control Ablation. We replace the control head with direct autoregressive generation of respond/silent decisions by the LLM, comparing it with our proposed response control mechanism.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
w/o Control Head67.431.169.530.4
Ours70.841.276.136.7

Results. Removing the control head leads to consistent performance degradation.

W1.1 TimeInteract is fine-tuned on STREAMTSI-34K and tested on its held-out split, and as far as I can tell none of the baselines is trained on this data..

(1) Established Evaluation Practice. Our evaluation follows a widely adopted practice in LLM research: fine-tuning TimeInteract on STREAMTSI-34K while evaluating all LLMs in a zero-shot setting. This protocol has also been adopted in recent LLM-based interaction studies [1–3]. Importantly, STREAMTSI-34K is itself a core contribution of our work, and using it to develop TimeInteract is therefore an essential part of our overall contribution rather than an unfair advantage.

[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.

[2] TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming. arXiv:2608.20958, 2026.

[3] Audio Interaction Model. arXiv:2606.05121, 2026.

(2) Stronger Baselines. In our experiments, we deliberately select substantially larger baselines (7B–14B), approximately 2–4× the size of our 4B backbone. These models inherently exhibit stronger general capabilities than our 4B backbone. Despite this gap, our 4B model achieves the best interaction performance through the proposed pipeline.

(3) Budget-Matched Training Comparison. To fully address your concern, we use the same 10% subset of STREAMTSI-34K and an identical training budget, applying streaming training to TimeInteract and full-parameter supervised fine-tuning (SFT) to selected LLM baselines for a controlled comparison.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Qwen2.5-7B-Instruct56.36.172.78.9
TimeOmni-144.79.464.29.6
TimeInteract (4B)70.841.276.136.7

Results. Under the same training data and computational budget, TimeInteract consistently outperforms both fine-tuned baselines across all four metrics.

W1.2: Apart from each baseline's input type, I couldn't find a description of how the baselines run in the streaming setting, such as their prompts and how the stream is presented to them.

a) Streaming Protocol.

(1) Streaming Evaluation Protocols. We consider two protocols for adapting existing LLMs to streaming time-series interaction:

ProtocolImplementationLimitation
Incremental Streaming (Adopted)Sequentially feed newly arrived chunks while retaining historical observations and interactions.Synchronous generation
Full-history Replay (Alternative)Re-input all accumulated chunks, user instructions, and model responses at each round.Exceeds the maximum context length as sequences grow

We adopt Incremental Streaming because, as time-series length increases, Full-history Replay exceeds the models' context limits for most samples. Similar evaluation protocols have also been adopted in recent studies [4,5].

[4] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.

[5] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.

(2) Additional Evaluation. We additionally evaluate full-history replay on the 50 shortest samples within the models' context limits. While this protocol is executable on these samples, it incurs substantially higher inference latency due to repeatedly processing the entire history, making it impractical for streaming applications.

Interaction Quality.

ProtocolModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Full-history ReplayQwen2.5-7B56.82.612.520.0
Qwen2.5-VL-7B46.715.412.520.0
TimeOmni-1-7B55.05.10.00.0
Incremental StreamingQwen2.5-7B49.22.60.00.0
Qwen2.5-VL-7B44.215.412.50.0
TimeOmni-1-7B48.30.00.00.0
Native StreamingTimeInteract (4B)61.756.425.040.0

Response Triggering.

ProtocolModelP ↑R ↑F1 ↑NQ-F1 ↑
Full-history ReplayQwen2.5-7B49.049.049.012.4
Qwen2.5-VL-7B21.841.428.68.3
TimeOmni-1-7B81.834.648.717.9
Incremental StreamingQwen2.5-7B95.844.260.53.6
Qwen2.5-VL-7B20.843.328.19.4
TimeOmni-1-7B75.535.648.414.3
Native StreamingTimeInteract (4B)92.481.786.772.3

Key Findings. (1) TimeInteract substantially outperforms all baselines. (2) Full-history replay yields mixed results and does not consistently outperform incremental streaming.

b) Baseline Prompt.

We have added the baseline prompt in Appendix C.6 (Evaluation Prompt Templates)‾\underline{\text{Appendix C.6 (Evaluation Prompt Templates)}} of the revised manuscript. We explicitly require baselines to output a respond or silent decision at each incoming chunk and generate an appropriate response when needed, considering current observations and preceding interaction history.

W1.3: The main tables cannot separate the architectural benefits from those of task-specific training. I suggest adding a baseline fine-tuned on the same data, ideally using the same backbone without the proposed components.

We address this concern through controlled architectural ablations (see W1) and budget-matched baseline fine-tuning (see W1.1, (3) Budget-Matched Training Comparison). Please refer to these sections for details.

W2: I'm also not fully comfortable with the evaluation, which is small and under-described.

a) Evaluation Scale.

We respectfully disagree that our evaluation uses a small number of episodes. On the contrary, 600 complete interaction episodes (400 ID + 200 OOD) represent a substantial evaluation scale. We spent significant API costs throughout the evaluation process. Our reasons are as follows:

(1) Differences Between Interaction Data and Traditional QA Data. Traditional QA samples require only one response. Our interaction episodes contain multiple chunks and require repeated response decisions. The evaluation includes 12,437 chunk-level decisions and 1,474 ground-truth responses. This represents a substantial evaluation workload.

StatisticsIDOODTotal
Episodes400200600
Single-turn / Multi-turn200 / 200100 / 100300 / 300
Total Chunks8,6783,75912,437
Ground-truth Responses1,0334411,474

(2) Larger Evaluation Scale Compared with Well-Established Studies. Our 600 interaction episodes represent a larger evaluation scale than comparable studies [1–5]. Notably, even influential studies, including those from NVIDIA [5] and JD.com [1], adopt evaluation scales comparable to or smaller than ours.

StudyEvaluation Samples
JoyAI-VL-Interaction [1]58
EgoVoice [2]170
OmniInteract [3]250
τ-Voice [4]278
StreamEval [5]300
TimeInteract (Ours)600

[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.

[2] EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams. EMNLP, 2026.

[3] OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants[J]. arXiv preprint arXiv:2605.26485, 2026.

[4] τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. ICML, 2026.

[5] StreamEval: Evaluating Interactive Streaming Video Generation Models. arXiv:2610.28402, 2026.

b) Dataset Detail.

(1) In-Domain (ID). Our ID evaluation includes 400 episodes, comprising both real-world (52.75%) and synthetic (47.25%) data, with 200 single-turn and 200 multi-turn episodes.

Data SourceSingle-turnMulti-turnTotalProportion
Real-world9112021152.8%
Synthetic1098018947.2%
Total200200400100%

(2) Out-of-Domain (OOD). Our OOD evaluation includes 200 episodes exclusively from real-world data, covering unseen domains and unseen data sources within domains.

DomainEpisodesProportion
Traffic5226.0%
Environment2512.5%
Healthcare2512.5%
Human Activity2512.5%
Agriculture2211.0%
Manufacturing2010.0%
Finance199.5%
Energy126.0%
Total200100.0%

W2.1: The paper doesn't say who or what assigns the semantic correctness score.

We thank the reviewer for pointing out this. Semantic scores are assigned by an LLM judge. The complete evaluation prompt is provided in Appendix C.6 (Evaluation Prompt Templates)‾\underline{\text{Appendix C.6 (Evaluation Prompt Templates)}} of the revised manuscript.

(1) Trigger Timing. During dataset construction, we annotate whether each chunk requires a response, providing ground-truth trigger labels. The corresponding trigger-timing evaluation results are reported in the main Table 2‾\underline{\text{Table 2}}. A response is considered correctly timed only when triggered at the annotated chunk.

(2) Response Correctness. For the main Table 1‾\underline{\text{Table 1}}, models that remain silent receive 0 by default, while generated responses are evaluated by an LLM judge based on their semantic quality and correctness.

Reviewer_7F5aOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates a framework for online time series tasks in which one is required to do inference continuously on some incoming stream of data. The build a dataset from some real world and synthetic time series and use LLMs to build a set of synthetic tasks to be performed on the corresponding time series. They categorize these tasks into four 'levels': L1: Understanding, L2: Persistence, L3: Initiative, L4: Adaptivity. These involve answering questions about the time series current state, defining a signal to monitor moving forward, pre-emptively warning a user about weird behaviour and modifying past behaviour of the LLM. They design an architecture to solve this problem using an LLM. Their architecture involves a local and complete historical normalization and embedding for the time series, along with taking in a user prompt. The model generates reasoning and planning based on the queries and series and ultimately produces a decision to produce a response or to wait (such as for a monitoring task) as well as a response / inference if deemed necessary. The system uses some kv cache forking and optimizations to handle the continual stream of data, changing user queries / tasks and determining weather to take an action or not and what the action would be. They also do some training to distill information produced while processing the time series so that it can be re-used in a recursive manner.

Critical strengths

The paper tackles many design decisions when attempting to build a continuous inference system for using LLMs to monitor time series.

The topic of using LLMs for continual time series monitoring is interesting.

Critical weaknesses

The experiments are not very convincing:

The goal of the work it feels to me is to adapt LLMs applied to time series (a growing field) to a setting where the LLM needs to continually monitor a time series and perform inference on it (the most likely setting for an LLM to be applied to time series, although evaluation and research typically does not cover the 'online' aspect). They produce an entire system designed to do this and then compare against 'non-streaming' approaches such as general LLMs etc. It is not explained at all how the 'general LLMs' are applied to this 'streaming setting'.

It appears that their proposed method requires finetuning the LLM used in their system but does not state whether any of the baselines are finetuned in any way, which would feel like an unfair comparison.

This work feels much less like it is tackling any research question but more that it is solving an engineering problem of building a specific solution to a particular problem. All of the aspects exist in other work: local and historical normalization / embedding, decoupling streami reading / embedding from writing / inference, kv cache forking, etc. I believe this style of work can be a valuable contribution for applied ML research, however the contribution is typically due to improving performance on some relevant task. In this case however, there is no motivating problem but rather a synthetic dataset built as an example of a type of problem that may show up in the real world. Further the experiments are not convincing that this system really improves performance. I am sure there exists anomaly detection datasets for example that one could've used as the basis for demonstrating their method. Ultimately given that all the ideas used to build their syste exist in other work and the application itself is not that compelling, it feels like their is very little contribution.

I think this work would be much better suited with the following motivation and contribution: Implementing LLMs for time series in practical environments requires continual monitoring, efficient processing of the series and answering of questions and the ability for the LLM to receive new and refined instructions as a way to account for new user preferences / tasks and allow the model to learn over the course of it's implementation. These aspects require many design considerations and a solid contribution would be experimenting with various options at each stage and determining a winning combination, validated on a practical problem.

Questions for authors

How are the other methods compared against (general LLMs, VLMs, TSLMs) adapted to the 'streaming setting'?

Are any of the other methods compared against finetuned in any way?

Can you give examples of any real applications? It seems most of the time series are synthetic and some are real but even for the real time series all the 'interaction tasks' are generated synthetically. The work would be better motivated if there was some real task being solved.

15 comments

We thank reviewer 7F5a for the review. We provide detailed point-by-point responses below and hope that our clarifications and additional evidence will fully address your concerns.

W1/Q1/Q2: Experimental Validation and Baseline Comparisons

W1: The experiments are not very convincing.

Q1: How are the other methods compared against (general LLMs, VLMs, TSLMs) adapted to the 'streaming setting'?

Q2: Are any of the other methods compared against finetuned in any way?

To further validate the effectiveness of TimeInteract and address concerns about baseline comparisons, we provide four additional sets of experiments:

  1. Streaming Protocol Comparison (See W4). We compare Incremental Streaming and Full-history Replay.
  2. Budget-Matched Fine-Tuning (See W5). We compare TimeInteract with fine-tuned LLM and TSLM baselines under the same training budget.
  3. Real-World OOD Evaluation (See W8). We evaluate interaction quality on unseen real-world time-series data.
  4. Controlled Architectural Ablations (W1). We compare dual-stream Mamba with single-stream Mamba, CNN, and Transformer encoders, and evaluate the response control head. We present the controlled architectural ablations below.

Setting. Given the computational cost of fine-tuning, we use the same 10% training subset and an identical training budget across all variants, including TimeInteract, to ensure a fair comparison.

a) Encoder Ablations. We consider two types of alternatives to the proposed dual-stream Mamba encoder:

(1) Single-stream Mamba. We replace the dual-stream encoder with a single-stream Mamba encoder to evaluate the benefit of dual-stream modeling.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Single-stream Mamba69.935.671.333.4
Ours70.841.276.136.7

Results. The dual-stream encoder outperforms single-stream Mamba across all four metrics.

(2) Alternative Encoders. We replace Mamba with CNN and Transformer encoders to examine the impact of the encoder architecture.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
CNN66.429.867.627.1
Transformer70.234.371.932.8
Ours70.841.276.136.7

Results. TimeInteract achieves better performance than both CNN and Transformer alternatives.

b) Response Control Ablation. We replace the control head with direct autoregressive generation of respond/silent decisions by the LLM, comparing it with our proposed response control mechanism.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
w/o Control Head67.431.169.530.4
Ours70.841.276.136.7

Results. Removing the control head leads to consistent performance degradation.

W2: The goal of the work it feels to me is to adapt LLMs applied to time series (a growing field) to a setting where the LLM needs to continually monitor a time series and perform inference on it (the most likely setting for an LLM to be applied to time series, although evaluation and research typically does not cover the 'online' aspect).

We do not consider this observation a weakness. Enabling LLMs to natively process streaming time-series data is precisely our research motivation. However, our goal extends beyond simply adapting existing time-series LLMs to streaming inputs. TimeInteract addresses continuous perception, timely response triggering, persistent monitoring, and adaptation to evolving user instructions. These capabilities are not jointly addressed by conventional time-series QA. Importantly, the significance of this research problem is also recognized by other reviewers:

  • Reviewer CWyr: "The paper addresses an important problem. ... useful capabilities that go beyond conventional time-series question answering."
  • Reviewer ofUB: "This paper takes a meaningful step toward real-world applications ..."
  • Reviewer w9nK: "I find the problem setup useful. ... concrete task levels that can be measured."

W3: They produce an entire system designed to do this and then compare against 'non-streaming' approaches such as general LLMs etc.

To the best of our knowledge, TimeInteract is the first model specifically designed for native streaming time-series interaction, and directly comparable models remain limited. Importantly, one of our primary goals is to establish a valuable new research problem and provide a foundation for future research.

Furthermore, adapting offline models for streaming evaluation is also an established practice in recent research. For example, [1]-[3] all evaluate conventional offline models under streaming evaluation protocols.

[1] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.

[2] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.

[3] OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? CVPR, 2025.

W4: It is not explained at all how the 'general LLMs' are applied to this 'streaming setting'.

(1) Streaming Evaluation Protocols. We consider two protocols for adapting existing LLMs to streaming time-series interaction:

ProtocolImplementationLimitation
Incremental Streaming (Adopted)Sequentially feed newly arrived chunks while retaining historical observations and interactions.Synchronous generation
Full-history Replay (Alternative)Re-input all accumulated chunks, user instructions, and model responses at each round.Exceeds the maximum context length as sequences grow

We adopt Incremental Streaming because, as time-series length increases, Full-history Replay exceeds the models' context limits for most samples. Similar evaluation protocols have also been adopted in recent studies [1,2].

[1] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.

[2] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.

(2) Additional Evaluation. We additionally evaluate full-history replay on the 50 shortest samples within the models' context limits. While this protocol is executable on these samples, it incurs substantially higher inference latency due to repeatedly processing the entire history, making it impractical for streaming applications.

Interaction Quality.

ProtocolModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Full-history ReplayQwen2.5-7B56.82.612.520.0
Qwen2.5-VL-7B46.715.412.520.0
TimeOmni-1-7B55.05.10.00.0
Incremental StreamingQwen2.5-7B49.22.60.00.0
Qwen2.5-VL-7B44.215.412.50.0
TimeOmni-1-7B48.30.00.00.0
Native StreamingTimeInteract (4B)61.756.425.040.0

Response Triggering.

ProtocolModelP ↑R ↑F1 ↑NQ-F1 ↑
Full-history ReplayQwen2.5-7B49.049.049.012.4
Qwen2.5-VL-7B21.841.428.68.3
TimeOmni-1-7B81.834.648.717.9
Incremental StreamingQwen2.5-7B95.844.260.53.6
Qwen2.5-VL-7B20.843.328.19.4
TimeOmni-1-7B75.535.648.414.3
Native StreamingTimeInteract (4B)92.481.786.772.3

Key Findings. (1) TimeInteract outperforms all baselines. (2) Full-history replay yields mixed results and does not consistently outperform incremental streaming.

W5: It appears that their proposed method requires finetuning the LLM used in their system but does not state whether any of the baselines are finetuned in any way, which would feel like an unfair comparison.

(1) Established Evaluation Practice. Our evaluation follows a widely adopted practice in LLM research: fine-tuning TimeInteract on STREAMTSI-34K while evaluating all LLMs in a zero-shot setting. This protocol has also been adopted in recent LLM-based interaction studies [1–3]. Importantly, STREAMTSI-34K is itself a core contribution of our work, and using it to develop TimeInteract is therefore an essential part of our overall contribution rather than an unfair advantage.

[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.

[2] TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming. arXiv:2608.20958, 2026.

[3] Audio Interaction Model. arXiv:2606.05121, 2026.

(2) Stronger Baselines. In our experiments, we deliberately select substantially larger baselines (7B–14B), approximately 2–4× the size of our 4B backbone. These models inherently exhibit stronger general capabilities than our 4B backbone. Despite this gap, our 4B model achieves the best interaction performance through the proposed pipeline.

(3) Budget-Matched Training Comparison. To fully address your concern, we use the same 10% subset of STREAMTSI-34K and an identical training budget, applying streaming training to TimeInteract and full-parameter supervised fine-tuning (SFT) to selected LLM baselines for a controlled comparison.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Qwen2.5-7B-Instruct56.36.172.78.9
TimeOmni-144.79.464.29.6
TimeInteract (4B)70.841.276.136.7

Results. Under the same training data and computational budget, TimeInteract consistently outperforms both fine-tuned baselines across all four metrics

W6: This work feels much less like it is tackling any research question but more that it is solving an engineering problem of building a specific solution to a particular problem. All of the aspects exist in other work: local and historical normalization / embedding, decoupling streaming reading / embedding from writing / inference, KV-cache forking, etc.

We respectfully disagree with this comment for the following reasons:

(1) Novel Research Problem. Our work introduces streaming time-series interaction as a new research problem, beyond conventional time-series QA. The research challenges lie in continuous perception, timely response triggering, persistent monitoring, and adaptation to evolving user instructions. Designing and evaluating mechanisms to address these challenges is precisely the research objective of TimeInteract, rather than merely an engineering implementation.

(2) Using Existing Techniques ≠ Lack of Innovation. Many influential time-series studies build upon existing techniques. For example, Time-MoE [1] combines existing Transformer and MoE techniques, Chronos [2] applies established language modeling techniques to time series, and ChatTS [3] leverages existing multimodal LLM techniques. All three build upon existing techniques, yet make valuable research contributions. The key criterion for innovation is the new insights a work brings to the time-series community.

[1] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. ICLR, 2025.

[2] Chronos: Learning the Language of Time Series. TMLR, 2024.

[3] ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning. VLDB, 2025.

W7: I believe this style of work can be a valuable contribution for applied ML research, however the contribution is typically due to improving performance on some relevant task.

We respectfully disagree that research contributions should be judged solely by performance improvements. Our contributions include (1) the first formulation of time-series interaction as a research problem, (2) a systematic interaction task framework and dataset, and (3) a specialized model for streaming time-series interaction.

Reviewer_ofUBOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Time-Series Interaction, a setting where models continuously process incoming time-series data, decide when to respond, and generate responses without interrupting stream processing. To address this setting, the authors propose TimeInteract, which combines a dual-view streaming encoder, a response control mechanism, and decoupled streaming inference. The paper also introduces StreamTSI-34K, a benchmark covering four interaction levels in single-turn and multi-turn settings. Experiments show that TimeInteract outperforms existing baselines across all four interaction levels, with better response triggering and lower inference latency.

Critical strengths
  • Although recent studies have explored online time-series forecasting [1], streaming language models [2], and real-time video interaction [3], much of time-series research still focuses on producing outputs from fixed input windows [4, 5]. This paper takes a meaningful step toward real-world applications by considering continuous monitoring and interaction with users. The proposed benchmark also provides a useful basis for evaluating these capabilities.

  • Inference efficiency is important for real-time time-series interaction. TimeInteract allows new observations to be processed during response generation, reducing inference latency compared with conventional approaches.

Critical weaknesses
  • The evaluation protocol for the baselines is not sufficiently described. In particular, it is unclear how time-series inputs and previous interactions are provided to each baseline, and how models without native streaming support determine when to respond. These details are important for assessing the comparisons in Tables 1 and 2.

  • The real-time feasibility of TimeInteract is not sufficiently evaluated. Although Table 3 reports low processing latency, it does not compare the processing time with the actual data arrival intervals. This makes it unclear whether the model can keep up with continuously arriving data across different sampling frequencies.

Questions for authors
  • How were time-series inputs, previous interactions, and response triggering handled for baselines without native streaming support? Could you clarify the evaluation protocol used in Tables 1 and 2?

  • Could you evaluate the maximum data arrival rate that TimeInteract can handle? In particular, how does its processing speed compare with the actual sampling frequencies of the datasets?

  • Could you report separate results for synthetic and real-world data to show whether the improvements across the four interaction tasks also hold on real-world time series?

[1] Lau et al., Fast and Slow Streams for Online Time Series Forecasting Without Information Leakage. ICLR 2025
[2] Tong et al., ProactiveLLM: Learning Active Interaction for Streaming Large Language Models. ICML 2026
[3] Zhang et al., Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models. CVPR 2026
[4] Nie et al., A Time Series Is Worth 64 Words: Long-Term Forecasting with Transformers. ICLR 2023
[5] Lee et al., TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting. NeurIPS 2025

5 comments

We appreciate reviewer ofUB for the thoughtful comments and insightful questions. We provide detailed point-by-point responses below.

W1/Q1: Baseline Evaluation Protocol

W1: The evaluation protocol for the baselines is not sufficiently described. In particular, it is unclear how time-series inputs and previous interactions are provided to each baseline, and how models without native streaming support determine when to respond.

Q1: How were time-series inputs, previous interactions, and response triggering handled for baselines without native streaming support?

(1) Streaming Evaluation Protocols.

We consider two protocols for adapting existing LLMs to streaming time-series interaction:

ProtocolImplementationLimitation
Incremental Streaming (Adopted)Sequentially feed newly arrived chunks while retaining historical observations and interactions.Synchronous generation
Full-history Replay (Alternative)Re-input all accumulated chunks, user instructions, and model responses at each round.Exceeds the maximum context length as sequences grow

We adopt Incremental Streaming because, as time-series length increases, Full-history Replay exceeds the models' context limits for most samples. Similar evaluation protocols have also been adopted in recent studies [1,2].

[1] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.

[2] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.

(2) Additional Evaluation. We additionally evaluate full-history replay on the 50 shortest samples within the models' context limits. While this protocol is executable on these samples, it incurs substantially higher inference latency due to repeatedly processing the entire history, making it impractical for streaming applications.

Interaction Quality.

ProtocolModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Full-history ReplayQwen2.5-7B56.82.612.520.0
Qwen2.5-VL-7B46.715.412.520.0
TimeOmni-1-7B55.05.10.00.0
Incremental StreamingQwen2.5-7B49.22.60.00.0
Qwen2.5-VL-7B44.215.412.50.0
TimeOmni-1-7B48.30.00.00.0
Native StreamingTimeInteract (4B)61.756.425.040.0

Response Triggering.

ProtocolModelP ↑R ↑F1 ↑NQ-F1 ↑
Full-history ReplayQwen2.5-7B49.049.049.012.4
Qwen2.5-VL-7B21.841.428.68.3
TimeOmni-1-7B81.834.648.717.9
Incremental StreamingQwen2.5-7B95.844.260.53.6
Qwen2.5-VL-7B20.843.328.19.4
TimeOmni-1-7B75.535.648.414.3
Native StreamingTimeInteract (4B)92.481.786.772.3

Key Findings. (1) TimeInteract outperforms all baselines. (2) Full-history replay yields mixed results and does not consistently outperform incremental streaming.

(3) How models without native streaming support determine when to respond.. During streaming evaluation, we explicitly require LLM baselines to output a respond or silent label at each incoming chunk. The label determines whether a response should be generated. If respond is selected, the model generates the corresponding response content; otherwise, it remains silent.

W2/Q2: Real-Time Processing Rate

W2: The real-time feasibility of TimeInteract is not sufficiently evaluated. Although Table 3 reports low processing latency, it does not compare the processing time with the actual data arrival intervals.

Q2: Could you evaluate the maximum data arrival rate that TimeInteract can handle? In particular, how does its processing speed compare with the actual sampling frequencies of the datasets?

(1) Real-Time Processing Rate.

Setting. We evaluate TimeInteract on a single NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB) without any engineering optimizations. We measure input processing rates under three scenarios: normal streaming, concurrent response generation, and triggering a new response.

Streaming Input Processing Rate (time steps/s).

VariablesScenarioP25MeanP75
<=2Normal Streaming7,1486,2465,580
Concurrent Response Generation4,3713,7543,356
New Response Triggered3,1092,7482,389
3–4Normal Streaming6,2465,8825,552
Concurrent Response Generation3,9823,5993,409
New Response Triggered2,9192,6732,328
>=5Normal Streaming5,3975,1474,857
Concurrent Response Generation3,5993,3473,157
New Response Triggered2,6112,4612,302

Results. TimeInteract achieves up to 7,148 time steps/s, maintaining over 3,300 time steps/s during concurrent response generation and over 2,300 time steps/s when triggering new responses.

(2) Comparison with Dataset Sampling Frequencies.

Our real-world datasets have sampling frequencies ranging from 0.13 Hz to 500 Hz. Even without any engineering optimization, TimeInteract achieves a minimum processing rate 4.6× higher than the highest sampling frequency.

Q3: Could you report separate results for synthetic and real-world data to show whether the improvements across the four interaction tasks also hold on real-world time series?

a) In-Domain (ID) Evaluation.

Setting. We separately evaluate interaction quality and response triggering on the real-world (211 episodes) and synthetic (189 episodes) subsets of our ID evaluation.

(1) Real-World Interaction Quality.

ModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct50.04.40.00.0
Qwen2.5-VL-7B41.60.021.10.0
TimeOmni-1-7B39.50.010.50.0
TimeInteract (4B)57.042.636.828.3

(2) Synthetic Interaction Quality.

ModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct43.50.02.30.0
Qwen2.5-VL-7B50.512.95.60.0
TimeOmni-1-7B42.20.00.00.0
TimeInteract (4B)77.756.541.692.3

(3) Real-World Response Triggering.

ModelP ↑R ↑F1 ↑NQ-F1 ↑
Qwen2.5-7B-Instruct71.765.668.65.0
Qwen2.5-VL-7B36.253.043.02.8
TimeOmni-1-7B84.644.057.910.2
TimeInteract (4B)87.884.686.254.6

(4) Synthetic Response Triggering.

ModelP ↑R ↑F1 ↑NQ-F1 ↑
Qwen2.5-7B-Instruct77.552.462.59.0
Qwen2.5-VL-7B26.360.936.711.2
TimeOmni-1-7B63.645.052.76.9
TimeInteract (4B)91.582.086.563.5

Results. (1) TimeInteract maintains strong performance on both real-world and synthetic data. (2) Interaction quality is generally higher on synthetic data than on real-world data, but response-triggering performance remains stable across both.

b) Out-of-Domain (OOD) Evaluation.

Setting. We additionally evaluate interaction quality on an OOD dataset consisting of 200 episodes entirely from unseen real-world data, covering unseen domains or data sources.

(1) OOD Real-World Interaction Quality.

ModelSizeIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct7B19.86.41.22.6
Qwen2.5-VL-7B7B9.20.07.30.0
TimeOmni-17B16.20.02.40.0
TimeInteract (4B)4B33.736.218.615.4

(2) OOD Response Triggering.

ModelSingle F1 ↑Single NQ-F1 ↑Multi F1 ↑Multi NQ-F1 ↑
Qwen2.5-7B-Instruct49.126.072.99.6
Qwen2.5-VL-7B14.18.519.99.7
TimeOmni-130.911.443.85.7
TimeInteract (4B)66.341.883.537.4

Results. TimeInteract outperforms all 7B baselines on unseen real-world data across all four tasks, retaining performance under domain shift.

AuthorsOpenReview ↗

Request of Reviewer's attention and feedback

Dear Reviewer,

As a gentle reminder, it has been more than three days since we submitted our rebuttal, and we were wondering whether our response has addressed your concerns. We look forward to your feedback and are prepared to address any further concerns you may have.

Thank you again for your valuable and constructive feedback, which has inspired further improvements to our paper.

Best regards,

The Authors

Reviewer_CWyrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies interactive language models for streaming time series and proposes TimeInteract, which combines a streaming time-series encoder, a learned mechanism for deciding when to respond, and separate control and response streams so that new observations can be processed during response generation. Plan tokens retain information about previous interactions without requiring the controller to wait for completed responses. The paper also introduces StreamTSI-34K, a dataset for relevant streaming time series tasks. The results show solid improvements over the selected baselines in interaction quality, response triggering, and latency.

Critical strengths
  • The paper addresses an important problem. Continuously monitoring a stream, remembering instructions, and deciding when to respond are useful capabilities that go beyond conventional time-series question answering.
  • The method is coherent and well motivated. The streaming encoder, response controller, and decoupled generation mechanism address complementary aspects of the problem.
  • The Plan-token mechanism is particularly interesting. It provides a way to retain interaction history while allowing stream processing to continue, and the distillation ablation provides meaningful evidence for its usefulness.
  • The dataset seems to be a substantial contribution. It covers several interaction types, includes both synthetic and real signals, and provides a useful foundation for studying this setting.
  • The reported improvements are strong and consistent across the selected comparisons. The additional out-of-domain evaluation and analyses of longer interactions strengthen the empirical study.
Critical weaknesses
  • The fairness and interpretation of the baseline comparisons are unclear. TimeInteract receives extensive task-specific training, but the paper does not sufficiently explain how the baselines are adapted to the same setting.
  • The evaluation protocol needs clearer specification. The paper defines the aggregate metrics, but leaves important questions about how response correctness, successful adaptation, and appropriate response timing are judged. These details are especially important for a new benchmark.
  • Most architectural ablations provide limited evidence about the design choices. Perturbing embeddings shows that the trained model uses them, but does not establish an advantage over simpler alternatives. The distillation ablation is a useful exception.
  • The real-time claims are stronger than the current efficiency evaluation supports. Reduced input blocking is valuable, but the latency measurements and workload description do not fully establish how well the system keeps up with sustained incoming data or longer streams.
  • Real-world generalization remains uncertain. The dataset is dominated by synthetic signals and constructed interactions. The out-of-domain results are helpful, but focus on response triggering and provide limited evidence about the quality of the resulting interactions.
  • The main results lack uncertainty estimates. The evaluation uses a relatively small sample of episodes, and the number of examples contributing to each task score is unclear.
Questions for authors
  • How were the baselines trained or adapted for streaming interaction, and how much of the improvement remains under comparable training?
  • How are response correctness, adaptation success, and trigger timing evaluated? Does multi-turn evaluation use each model’s own previous responses?
  • How does performance differ between synthetic and real signals, and what exactly is held out in the out-of-domain evaluation?
  • How much do the main architectural components help compared with simpler, appropriately trained alternatives?
  • How does the system behave over longer streams or when new observations and user instructions arrive during response generation?
6 comments

We thank the reviewer CWyr for the valuable and constructive feedback. We have carefully considered all the concerns and provide point-by-point responses below.

W1/Q1: Baseline Fairness & Streaming Protocol

W1: The fairness and interpretation of the baseline comparisons are unclear. ... the paper does not sufficiently explain how the baselines are adapted to the same setting.

Q1: How were the baselines trained or adapted for streaming interaction, ... under comparable training?

a) Baseline Fairness.

(1) Established Evaluation Practice. Our evaluation follows a widely adopted practice in LLM research: fine-tuning TimeInteract on STREAMTSI-34K while evaluating all LLMs in a zero-shot setting. This protocol has also been adopted in recent LLM-based interaction studies [1–3]. Importantly, STREAMTSI-34K is itself a core contribution of our work, and using it to develop TimeInteract is therefore an essential part of our overall contribution rather than an unfair advantage.

[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.

[2] TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming. arXiv:2608.20958, 2026.

[3] Audio Interaction Model. arXiv:2606.05121, 2026.

(2) Stronger Baselines. In our experiments, we deliberately select substantially larger baselines (7B–14B), approximately 2–4× the size of our 4B backbone. These models inherently exhibit stronger general capabilities than our 4B backbone. Despite this gap, our 4B model achieves the best interaction performance through the proposed pipeline.

(3) Budget-Matched Training Comparison. To fully address your concern, we use the same 10% subset of STREAMTSI-34K and an identical training budget, applying streaming training to TimeInteract and full-parameter supervised fine-tuning (SFT) to selected LLM baselines for a controlled comparison.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Qwen2.5-7B-Instruct56.36.172.78.9
TimeOmni-144.79.464.29.6
TimeInteract (4B)70.841.276.136.7

Results. Under the same training data and computational budget, TimeInteract consistently outperforms both fine-tuned baselines across all four metrics

b) Streaming Protocol.

(1) Streaming Evaluation Protocols. We consider two protocols for adapting existing LLMs to streaming time-series interaction:

ProtocolImplementationLimitation
Incremental Streaming (Adopted)Feed new chunks sequentially, retaining observations and interactions.Synchronous generation
Full-history Replay (Alternative)Re-input all chunks, instructions, and responses each round.Context overflow on long streams

We adopt Incremental Streaming because, as time-series length increases, Full-history Replay exceeds the models' context limits for most samples. Similar evaluation protocols have also been adopted in recent studies [4,5].

[4] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.

[5] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.

(2) Additional Evaluation. We additionally evaluate full-history replay on the 50 shortest samples within the models' context limits. While this protocol is executable on these samples, it incurs substantially higher inference latency due to repeatedly processing the entire history, making it impractical for streaming applications.

Interaction Quality.

ProtocolModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Full-history ReplayQwen2.5-7B56.82.612.520.0
Qwen2.5-VL-7B46.715.412.520.0
TimeOmni-1-7B55.05.10.00.0
Incremental StreamingQwen2.5-7B49.22.60.00.0
Qwen2.5-VL-7B44.215.412.50.0
TimeOmni-1-7B48.30.00.00.0
Native StreamingTimeInteract (4B)61.756.425.040.0

Response Triggering.

ProtocolModelP ↑R ↑F1 ↑NQ-F1 ↑
Full-history ReplayQwen2.5-7B49.049.049.012.4
Qwen2.5-VL-7B21.841.428.68.3
TimeOmni-1-7B81.834.648.717.9
Incremental StreamingQwen2.5-7B95.844.260.53.6
Qwen2.5-VL-7B20.843.328.19.4
TimeOmni-1-7B75.535.648.414.3
Native StreamingTimeInteract (4B)92.481.786.772.3

Key Findings. (1) TimeInteract outperforms all baselines. (2) Full-history replay yields mixed results and does not consistently outperform incremental streaming.

W2/Q2: Evaluation Details

W2: The evaluation protocol needs clearer specification. The paper defines the aggregate metrics, but leaves important questions about how response correctness, successful adaptation, and appropriate response timing are judged. These details are especially important for a new benchmark.

Q2: How are response correctness, adaptation success, and trigger timing evaluated? Does multi-turn evaluation use each model’s own previous responses?

a) Evaluation Criteria.

(1) Trigger Timing. During dataset construction, we annotate whether each chunk requires a response, providing ground-truth trigger labels. The corresponding trigger-timing evaluation results are reported in the main Table 2‾\underline{\text{Table 2}}. A response is considered correctly timed only when triggered at the annotated chunk.

(2) Response Correctness. For the main Table 1‾\underline{\text{Table 1}}, models that remain silent receive 0 by default, while generated responses are evaluated by an LLM judge based on their semantic quality and correctness.

(3) Adaptation Success. Similar to response correctness, models that remain silent receive 0 by default, while generated responses are evaluated by an LLM judge for adherence to updated user instructions and preferences.

b) Does multi-turn evaluation use each model’s own previous responses?

Yes. During evaluation, the LLM judge jointly considers the preceding interaction history and the ground-truth answer corresponding to the current response when assessing response quality. We have added the corresponding evaluation prompt in Appendix C.6 (Evaluation Prompt Templates)‾\underline{\text{Appendix C.6 (Evaluation Prompt Templates)}} of the revised manuscript.

Q3: How does performance differ between synthetic and real signals, and what exactly is held out in the out-of-domain evaluation?

a) Evaluation Data Details.

(1) In-Domain (ID). Our ID evaluation includes 400 episodes, comprising both real-world (52.75%) and synthetic (47.25%) data, with 200 single-turn and 200 multi-turn episodes.

Data SourceSingle-turnMulti-turnTotalProportion
Real-world9112021152.8%
Synthetic1098018947.2%
Total200200400100%

(2) Out-of-Domain (OOD). Our OOD evaluation includes 200 episodes exclusively from real-world data, covering unseen domains and unseen data sources within domains.

DomainEpisodesProportion
Traffic5226.0%
Environment2512.5%
Healthcare2512.5%
Human Activity2512.5%
Agriculture2211.0%
Manufacturing2010.0%
Finance199.5%
Energy126.0%
Total200100.0%

b) Synthetic vs. Real-World Performance.

Setting. We separately evaluate interaction quality and response triggering on the real-world (211 episodes) and synthetic (189 episodes) subsets of our ID evaluation.

(1) Real-World Interaction Quality.

ModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct50.04.40.00.0
Qwen2.5-VL-7B41.60.021.10.0
TimeOmni-1-7B39.50.010.50.0
TimeInteract (4B)57.042.636.828.3

(2) Synthetic Interaction Quality.

ModelIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct43.50.02.30.0
Qwen2.5-VL-7B50.512.95.60.0
TimeOmni-1-7B42.20.00.00.0
TimeInteract (4B)77.756.541.692.3

(3) Real-World Response Triggering.

ModelP ↑R ↑F1 ↑NQ-F1 ↑
Qwen2.5-7B-Instruct71.765.668.65.0
Qwen2.5-VL-7B36.253.043.02.8
TimeOmni-1-7B84.644.057.910.2
TimeInteract (4B)87.884.686.254.6

(4) Synthetic Response Triggering.

ModelP ↑R ↑F1 ↑NQ-F1 ↑
Qwen2.5-7B-Instruct77.552.462.59.0
Qwen2.5-VL-7B26.360.936.711.2
TimeOmni-1-7B63.645.052.76.9
TimeInteract (4B)91.582.086.563.5

Results. (1) TimeInteract maintains strong performance on both real-world and synthetic data. (2) Interaction quality is generally higher on synthetic data than on real-world data, but response-triggering performance remains stable across both.

W3/Q4: Architectural Ablations

W3: Most architectural ablations provide limited evidence about the design choices. Perturbing embeddings shows that the trained model uses them, but does not establish an advantage over simpler alternatives. The distillation ablation is a useful exception.

Q4: How much do the main architectural components help compared with simpler, appropriately trained alternatives?

We agree that current architectural ablations leave some comparisons unexplored, particularly regarding the encoder architecture and response control mechanism. We therefore add two groups of controlled comparisons. Given the computational cost of fine-tuning, we use the same 10% training subset and an identical training budget across all variants, including TimeInteract, to ensure a fair comparison.

a) Encoder Ablations. We consider two alternatives to the proposed dual-stream Mamba encoder:

(1) Single-stream Mamba. We replace the dual-stream encoder with a single-stream Mamba encoder to evaluate the benefit of dual-stream modeling.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
Single-stream Mamba69.935.671.333.4
Ours70.841.276.136.7

Results. The dual-stream encoder outperforms single-stream Mamba across all four metrics.

(2) Alternative Encoders. We replace Mamba with CNN and Transformer encoders to examine the impact of the encoder architecture.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
CNN66.429.867.627.1
Transformer70.234.371.932.8
Ours70.841.276.136.7

Results. TimeInteract achieves better performance than both CNN and Transformer alternatives.

b) Response Control Ablation. We replace the control head with direct autoregressive generation of respond/silent decisions by the LLM, comparing it with our proposed response control mechanism.

ModelSingle-turn F1 ↑Single-turn NQ-F1 ↑Multi-turn F1 ↑Multi-turn NQ-F1 ↑
w/o Control Head67.431.169.530.4
Ours70.841.276.136.7

Results. Removing the control head leads to consistent performance degradation.

W4/Q5: Real-Time Efficiency

W4: The real-time claims are stronger than the current efficiency evaluation supports. Reduced input blocking is valuable, but the latency measurements and workload description do not fully establish how well the system keeps up with sustained incoming data or longer streams.

Q5: How does the system behave over longer streams or when new observations and user instructions arrive during response generation?

Setting. We evaluate TimeInteract on a single NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB) without any engineering optimizations. We measure input processing under three scenarios: normal streaming, concurrent response generation, and triggering a new response.

Streaming Input Processing Rate (time steps/s).

VariablesScenarioP25MeanP75
<=2Normal Streaming7,1486,2465,580
Concurrent Response Generation4,3713,7543,356
New Response Triggered3,1092,7482,389
3–4Normal Streaming6,2465,8825,552
Concurrent Response Generation3,9823,5993,409
New Response Triggered2,9192,6732,328
>=5Normal Streaming5,3975,1474,857
Concurrent Response Generation3,5993,3473,157
New Response Triggered2,6112,4612,302

Results. TimeInteract maintains over 3,300 time steps/s during concurrent response generation and over 2,300 time steps/s when triggering new responses, demonstrating efficient input processing even without engineering optimizations.

W5: Real-world generalization remains uncertain. The dataset is dominated by synthetic signals and constructed interactions. The out-of-domain results are helpful, but focus on response triggering and provide limited evidence about the quality of the resulting interactions.

Setting. We evaluate interaction quality on an OOD dataset consisting entirely of unseen real-world data (see Q3 for dataset details).

ModelSizeIQA SC ↑PIF IFR ↑PTW CEHR ↑UGA ASR ↑
Qwen2.5-7B-Instruct7B19.86.41.22.6
Qwen2.5-VL-7B7B9.20.07.30.0
TimeOmni-17B16.20.02.40.0
TimeInteract (4B)4B33.736.218.615.4

Results. Even on the OOD dataset, TimeInteract outperforms 7B baselines in interaction quality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their insightful reviews and valuable comments, which have helped us further improve our paper.

This paper introduces Time-Series Interaction as a new research problem and develops TimeInteract, a specialized model explicitly designed for streaming time-series interaction. It further establishes a systematic interaction task framework and constructs the STREAMTSI-34K dataset. Extensive evaluations validate the model's effectiveness across diverse interaction tasks.

Most reviewers recognized the significance of our work, describing the research problem as "important" (Reviewer CWyr) and "useful" (Reviewer w9nK). The method was considered "coherent and well motivated" (Reviewer CWyr), while the work was recognized as "a meaningful step toward real-world applications" (Reviewer ofUB).

The reviewers also raised insightful and constructive concerns. We have provided additional experiments and detailed clarifications to address them. Below is a summary of our responses:

  • Clarify streaming evaluation protocols (CWyr, ofUB, w9nK, 7F5a): We clarify how existing LLMs, VLMs, and TSLMs are adapted to streaming interaction, including input handling and response triggering. We further compare Incremental Streaming and Full-history Replay.

  • Add budget-matched fine-tuning comparisons (CWyr, w9nK, 7F5a): We compare TimeInteract with fine-tuned LLM and TSLM baselines under the same training data and computational budget.

  • Provide controlled architectural ablations (CWyr, w9nK, 7F5a): We compare dual-stream Mamba with single-stream Mamba, CNN, and Transformer encoders, and evaluate the contribution of the response control head.

  • Evaluate real-time processing efficiency (CWyr, ofUB): We report input processing rates under normal streaming, concurrent response generation, and new response triggering, and compare them with real-world sampling frequencies.

  • Extend real-world and OOD evaluations (CWyr, ofUB, 7F5a): We provide separate real-world and synthetic evaluation results, alongside additional OOD interaction-quality evaluations on unseen real-world time series.

  • Clarify evaluation details and reliability (CWyr, w9nK): We clarify evaluation criteria and prompts, and report dataset composition, per-task test counts, and repeated scoring results.

  • Discuss practical applications and limitations (w9nK, 7F5a): We present interaction examples based on real-world time-series datasets, clarify how constructed interactions are grounded in real-world observations and annotations, and acknowledge the limitations of constructed interaction scenarios.

To address these concerns, we invested more than 200 GPU-hours on NVIDIA RTX PRO 6000 GPUs (96 GB), along with substantial API costs, to conduct additional experiments and evaluations.

The reviewers' valuable suggestions have been greatly helpful in improving our paper. We would be happy to answer any further questions.

We look forward to the reviewers' feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.