- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies interaction with a streaming time series, where the model decides on its own when to respond and keeps reading new data while it generates. It makes three contributions.
- TimeInteract builds on a small instruction-tuned LLM and adds a fast/slow stream encoder, a control head with compact Plan tokens that summarize earlier replies, and a forked KV cache that lets generation run without blocking the stream.
- A four-level task hierarchy covers answering questions, following standing instructions, warning proactively and adapting to user feedback.
- STREAMTSI-34K is a dataset built from synthetic and real series, with interactions constructed from templates, rules and LLM generation and then checked by an LLM verifier.
TimeInteract is trained on this dataset and, on its held-out split, scores highest on every task metric and on response-triggering F1 against a range of existing LLMs, vision-language models and time-series LMs. It also completes responses faster than interleaved generation.
Critical strengths
- I find the problem setup useful. The paper turns streaming interaction, where the model has to decide when to respond while data keeps arriving, into concrete task levels that can be measured.
- The efficiency results support the decoupled inference design, and I think it is the most solid part of the paper. Forking the KV cache brings the stall during generation close to zero and shortens completion time relative to interleaved generation, especially for longer responses. Time to first token stays about the same.
Critical weaknesses
- My main concern is that the main comparison can't isolate the method.
- TimeInteract is fine-tuned on STREAMTSI-34K and tested on its held-out split, and as far as I can tell none of the baselines is trained on this data.
- Apart from each baseline's input type, I couldn't find a description of how the baselines run in the streaming setting, such as their prompts and how the stream is presented to them.
- Because of this, I don't think the main tables can separate the benefit of the architecture from the benefit of training on the benchmark's own format. I'd suggest adding a baseline fine-tuned on the same data, ideally the same backbone without the proposed components.
- I'm also not fully comfortable with the evaluation, which is small and under-described.
- The paper doesn't say who or what assigns the semantic correctness score.
- The main tables come from a small test sample, and they report no variance estimates, no repeated runs and no per-task test counts.
- The test labels come from the same construction pipeline as the training data, which relies on LLM generation and LLM verification, and humans review only the low-confidence cases, so I'd be careful about reading the scores as measures of real interaction quality.
Questions for authors
- How exactly is each baseline run in the streaming setting? Have you fine-tuned any baseline on STREAMTSI-34K?
- How is semantic correctness scored? If an LLM judge is used, which model and prompt?
- How many test items does each task have, and how much do the results vary across training seeds?
8 comments
W1.2: Apart from each baseline's input type, I couldn't find a description of how the baselines run in the streaming setting, such as their prompts and how the stream is presented to them.
a) Streaming Protocol.
(1) Streaming Evaluation Protocols. We consider two protocols for adapting existing LLMs to streaming time-series interaction:
| Protocol | Implementation | Limitation |
|---|---|---|
| Incremental Streaming (Adopted) | Sequentially feed newly arrived chunks while retaining historical observations and interactions. | Synchronous generation |
| Full-history Replay (Alternative) | Re-input all accumulated chunks, user instructions, and model responses at each round. | Exceeds the maximum context length as sequences grow |
We adopt Incremental Streaming because, as time-series length increases, Full-history Replay exceeds the models' context limits for most samples. Similar evaluation protocols have also been adopted in recent studies [4,5].
[4] From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios. arXiv:2610.21568, 2026.
[5] StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video. arXiv:2605.16381, 2026.
(2) Additional Evaluation. We additionally evaluate full-history replay on the 50 shortest samples within the models' context limits. While this protocol is executable on these samples, it incurs substantially higher inference latency due to repeatedly processing the entire history, making it impractical for streaming applications.
Interaction Quality.
| Protocol | Model | IQA SC ↑ | PIF IFR ↑ | PTW CEHR ↑ | UGA ASR ↑ |
|---|---|---|---|---|---|
| Full-history Replay | Qwen2.5-7B | 56.8 | 2.6 | 12.5 | 20.0 |
| Qwen2.5-VL-7B | 46.7 | 15.4 | 12.5 | 20.0 | |
| TimeOmni-1-7B | 55.0 | 5.1 | 0.0 | 0.0 | |
| Incremental Streaming | Qwen2.5-7B | 49.2 | 2.6 | 0.0 | 0.0 |
| Qwen2.5-VL-7B | 44.2 | 15.4 | 12.5 | 0.0 | |
| TimeOmni-1-7B | 48.3 | 0.0 | 0.0 | 0.0 | |
| Native Streaming | TimeInteract (4B) | 61.7 | 56.4 | 25.0 | 40.0 |
Response Triggering.
| Protocol | Model | P ↑ | R ↑ | F1 ↑ | NQ-F1 ↑ |
|---|---|---|---|---|---|
| Full-history Replay | Qwen2.5-7B | 49.0 | 49.0 | 49.0 | 12.4 |
| Qwen2.5-VL-7B | 21.8 | 41.4 | 28.6 | 8.3 | |
| TimeOmni-1-7B | 81.8 | 34.6 | 48.7 | 17.9 | |
| Incremental Streaming | Qwen2.5-7B | 95.8 | 44.2 | 60.5 | 3.6 |
| Qwen2.5-VL-7B | 20.8 | 43.3 | 28.1 | 9.4 | |
| TimeOmni-1-7B | 75.5 | 35.6 | 48.4 | 14.3 | |
| Native Streaming | TimeInteract (4B) | 92.4 | 81.7 | 86.7 | 72.3 |
Key Findings. (1) TimeInteract substantially outperforms all baselines. (2) Full-history replay yields mixed results and does not consistently outperform incremental streaming.
b) Baseline Prompt.
We have added the baseline prompt in of the revised manuscript. We explicitly require baselines to output a respond or silent decision at each incoming chunk and generate an appropriate response when needed, considering current observations and preceding interaction history.
W1.3: The main tables cannot separate the architectural benefits from those of task-specific training. I suggest adding a baseline fine-tuned on the same data, ideally using the same backbone without the proposed components.
We address this concern through controlled architectural ablations (see W1) and budget-matched baseline fine-tuning (see W1.1, (3) Budget-Matched Training Comparison). Please refer to these sections for details.
W2: I'm also not fully comfortable with the evaluation, which is small and under-described.
a) Evaluation Scale.
We respectfully disagree that our evaluation uses a small number of episodes. On the contrary, 600 complete interaction episodes (400 ID + 200 OOD) represent a substantial evaluation scale. We spent significant API costs throughout the evaluation process. Our reasons are as follows:
(1) Differences Between Interaction Data and Traditional QA Data. Traditional QA samples require only one response. Our interaction episodes contain multiple chunks and require repeated response decisions. The evaluation includes 12,437 chunk-level decisions and 1,474 ground-truth responses. This represents a substantial evaluation workload.
| Statistics | ID | OOD | Total |
|---|---|---|---|
| Episodes | 400 | 200 | 600 |
| Single-turn / Multi-turn | 200 / 200 | 100 / 100 | 300 / 300 |
| Total Chunks | 8,678 | 3,759 | 12,437 |
| Ground-truth Responses | 1,033 | 441 | 1,474 |
(2) Larger Evaluation Scale Compared with Well-Established Studies. Our 600 interaction episodes represent a larger evaluation scale than comparable studies [1–5]. Notably, even influential studies, including those from NVIDIA [5] and JD.com [1], adopt evaluation scales comparable to or smaller than ours.
| Study | Evaluation Samples |
|---|---|
| JoyAI-VL-Interaction [1] | 58 |
| EgoVoice [2] | 170 |
| OmniInteract [3] | 250 |
| τ-Voice [4] | 278 |
| StreamEval [5] | 300 |
| TimeInteract (Ours) | 600 |
[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.
[2] EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams. EMNLP, 2026.
[3] OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants[J]. arXiv preprint arXiv:2605.26485, 2026.
[4] τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. ICML, 2026.
[5] StreamEval: Evaluating Interactive Streaming Video Generation Models. arXiv:2610.28402, 2026.
b) Dataset Detail.
(1) In-Domain (ID). Our ID evaluation includes 400 episodes, comprising both real-world (52.75%) and synthetic (47.25%) data, with 200 single-turn and 200 multi-turn episodes.
| Data Source | Single-turn | Multi-turn | Total | Proportion |
|---|---|---|---|---|
| Real-world | 91 | 120 | 211 | 52.8% |
| Synthetic | 109 | 80 | 189 | 47.2% |
| Total | 200 | 200 | 400 | 100% |
(2) Out-of-Domain (OOD). Our OOD evaluation includes 200 episodes exclusively from real-world data, covering unseen domains and unseen data sources within domains.
| Domain | Episodes | Proportion |
|---|---|---|
| Traffic | 52 | 26.0% |
| Environment | 25 | 12.5% |
| Healthcare | 25 | 12.5% |
| Human Activity | 25 | 12.5% |
| Agriculture | 22 | 11.0% |
| Manufacturing | 20 | 10.0% |
| Finance | 19 | 9.5% |
| Energy | 12 | 6.0% |
| Total | 200 | 100.0% |
W2.1: The paper doesn't say who or what assigns the semantic correctness score.
We thank the reviewer for pointing out this. Semantic scores are assigned by an LLM judge. The complete evaluation prompt is provided in of the revised manuscript.
(1) Trigger Timing. During dataset construction, we annotate whether each chunk requires a response, providing ground-truth trigger labels. The corresponding trigger-timing evaluation results are reported in the main . A response is considered correctly timed only when triggered at the annotated chunk.
(2) Response Correctness. For the main , models that remain silent receive 0 by default, while generated responses are evaluated by an LLM judge based on their semantic quality and correctness.
We thank the reviewer w9nK for the careful assessment and constructive suggestions. We provide detailed point-by-point responses below and hope that our clarifications and additional evidence will fully address your concerns.
To better isolate the contributions of our proposed architecture, we add two groups of controlled architectural ablations. All variants, including TimeInteract, are trained on the same 10% subset of STREAMTSI-34K with an identical training budget to ensure fair comparisons.
a) Encoder Ablations. We consider two alternatives to the proposed dual-stream Mamba encoder:
(1) Single-stream Mamba. We replace the dual-stream encoder with a single-stream Mamba encoder to evaluate the benefit of dual-stream modeling.
Results. The dual-stream encoder outperforms single-stream Mamba across all four metrics.
(2) Alternative Encoders. We replace Mamba with CNN and Transformer encoders to examine the impact of the encoder architecture.
Results. TimeInteract achieves better performance than both CNN and Transformer alternatives.
b) Response Control Ablation. We replace the control head with direct autoregressive generation of
respond/silentdecisions by the LLM, comparing it with our proposed response control mechanism.Results. Removing the control head leads to consistent performance degradation.
(1) Established Evaluation Practice. Our evaluation follows a widely adopted practice in LLM research: fine-tuning TimeInteract on STREAMTSI-34K while evaluating all LLMs in a zero-shot setting. This protocol has also been adopted in recent LLM-based interaction studies [1–3]. Importantly, STREAMTSI-34K is itself a core contribution of our work, and using it to develop TimeInteract is therefore an essential part of our overall contribution rather than an unfair advantage.
[1] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence. arXiv:2606.14777, 2026.
[2] TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming. arXiv:2608.20958, 2026.
[3] Audio Interaction Model. arXiv:2606.05121, 2026.
(2) Stronger Baselines. In our experiments, we deliberately select substantially larger baselines (7B–14B), approximately 2–4× the size of our 4B backbone. These models inherently exhibit stronger general capabilities than our 4B backbone. Despite this gap, our 4B model achieves the best interaction performance through the proposed pipeline.
(3) Budget-Matched Training Comparison. To fully address your concern, we use the same 10% subset of STREAMTSI-34K and an identical training budget, applying streaming training to TimeInteract and full-parameter supervised fine-tuning (SFT) to selected LLM baselines for a controlled comparison.
Results. Under the same training data and computational budget, TimeInteract consistently outperforms both fine-tuned baselines across all four metrics.