acceptodds
Under review as a conference paper at ICLR 2027

Deep Think with Rehearsal for Low-Latency Team-AI Collaboration

Abstract

The integration of Large Language Models (LLMs) into scientific team meetings presents exciting opportunities to accelerate biomedical discovery, especially through their strong deep thinking capabilities enabled by multi-step reasoning and web search. However, such methods are computationally expensive and introduce substantial latency, limiting their effectiveness in real-time Team-AI communications. In this study, we propose Deep Think with Rehearsal (DTR), a novel framework that decouples deep reasoning from synchronous interaction in the AI4Science context. DTR transfers computationally intensive reasoning into an offline rehearsal phase, allowing the LLM to pre-cognize complex scientific contexts and deliver high-quality, "deep" insights with minimal latency during live interactions. To facilitate this research, we introduce the Scientific Team Meeting Dataset (STMD), a hybrid benchmark comprising authentic transcripts from three real-world biomedical research labs alongside extensive simulated multi-party deliberations synthesized from PubMed literature. Experiments in both simulated and real-world settings demonstrate that DTR consistently improves response quality while reducing inference latency compared to state-of-the-art methods, highlighting the effectiveness of rehearsal in enabling low-latency, high-quality scientific collaboration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Human-AI Collaboration, Large Language Models, LLM reasoning, Low-Latency Interaction, Biomedical NLP, Tool-Augmented Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xSTVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method, "Deep Think with Rehearsal," to decouple the reasoning from the real-time interaction. They find that the method can greatly reduce the time for first token response, and increase the quality of the response. Their quantitative results show both scores from ROUGE-1 and human evaluation from 200 feedback instances.

Critical strengths

This is an interesting and clean idea motivated by a real-world setting. The positive results also converge with the intuition that 'reading in advance' will help the retrieval in real time more than 'reading on the fly'.

Critical weaknesses

The quantitative method needs more rigorous evaluation. The ROUGE-1/BLEU score is not a good metric for this task as it misses the semantic information. The human evaluation is also not convincing, as many technical details are underspecified; for instance, the distribution info is missing, and there are no error bars.

Questions for authors

A previous paper (arXiv:2504.13171) seems to propose a similar idea, yet the authors did not cite it. Can the authors comment on this prior work and clarify the novelty of their method?

Reviewer_7z5COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Deep Think with Rehearsal (DTR), a framework for enabling low-latency LLM assistance in scientific team meetings by shifting expensive reasoning and external knowledge retrieval from online interaction to offline preparation.

DTR simulates multi-role scientific discussions using team memory and role-specific personas, applies confidence-based filtering to select simulated trajectories, and uses a gating model to selectively trigger deeper reasoning and knowledge acquisition. The resulting knowledge notes are stored for lightweight retrieval during actual meetings.

The authors introduce the Scientific Team Meeting Dataset (STMD), combining PubMed-grounded synthetic meetings with transcripts from three biomedical research teams.

Experiments across multiple LLM backbones report improvements in response quality and substantial reductions in online latency compared with several online reasoning and retrieval baselines. Human-involved evaluations also report higher quality, helpfulness, and personalization ratings.

Critical strengths

S1. The paper is trying to address a practically relevant latency challenge in real-time human-AI collaboration.

The tension between deep reasoning quality and interactive response latency is meaningful, especially in scientific collaboration where substantial external knowledge and multi-step reasoning may be required. Shifting expensive computations offline is a practical system design choice. The reported results demonstrate substantial online latency reductions across multiple backbone models, suggesting potential usefulness in sufficiently predictable and well-scoped applications.

S2. The paper makes an effort to evaluate the proposed framework in a domain-specific collaborative setting.

The STMD benchmark combines large-scale synthetic multi-party discussions with meeting transcripts collected from real biomedical research teams. The inclusion of component ablations, offline computation analysis, and human-involved evaluations is valuable, although important limitations in the evaluation design remain.

Critical weaknesses

W1. The fundamental premise of rehearsal-based scientific collaboration is insufficiently justified, raising concerns about the problem formulation and practical significance.

  • W1.1. Topic predictability does not imply discussion predictability. The paper assumes that scientific meetings are sufficiently predictable because they revolve around predefined research objectives, datasets, and hypotheses (Introduction). However, knowing the general meeting topic is fundamentally different from anticipating the specific questions, disagreements, unexpected findings, and emerging hypotheses that determine the actual course of scientific discussions. A single unexpected observation or challenge to an underlying assumption can redirect a meeting toward an entirely new topic that was not foreseeable from the initial agenda.

  • W1.2. The usefulness and coverage of offline rehearsal remain unestablished. The central motivation of DTR relies on simulated meetings anticipating future information needs. However, the paper provides no direct evidence that these simulations cover a meaningful proportion of the important questions arising in actual scientific meetings. Even realistic and coherent simulated discussions may differ substantially from real discussions in terms of what information becomes necessary. This simulation-to-reality gap is particularly consequential because the value of scientific collaboration often comes from unanticipated insights rather than predictable exchanges.

W1.3. The claimed benefit of rehearsal is therefore conditional on an unverified assumption. If discussions largely follow predictable trajectories, conventional pre-meeting literature review and knowledge preparation may already be sufficient. Conversely, if high-value discussions frequently involve unexpected directions, the precomputed knowledge may have limited relevance and the system may still require expensive online reasoning. The paper does not establish the extent to which real scientific meetings fall into the regime where rehearsal is both necessary and effective. This undermines the central motivation rather than merely presenting an implementation limitation.

W2. The benchmark construction and experimental design do not convincingly validate the key claims of the paper.

W2.1. Retrospective benchmark construction may introduce artificial predictability. STMD-Sim is constructed from published PubMed papers using a Prophet LLM with access to the full paper, including its final findings. The extracted golden conclusions and topic sequences are then used to guide simulated discussions, with the speaker-selection prompt explicitly conditioned on these artifacts (Section 4 and Appendix F). The evaluation subsequently measures similarity between model responses and reference answers derived from the same papers' conclusions. This setup risks creating an artificially structured discussion environment where trajectories are guided toward known outcomes. As a result, strong ROUGE/BLEU performance may reflect reconstruction of predetermined scientific conclusions rather than the ability to anticipate genuinely unknown information needs in real scientific collaboration.

W2.2. The evaluation does not directly measure the capability that motivates DTR. The core claim is that offline rehearsal anticipates future reasoning needs and enables useful, low-latency responses during real discussions. Yet the experiments primarily evaluate answer similarity and response latency, without directly measuring rehearsal knowledge coverage, the relevance of simulated trajectories to actual meeting trajectories, or performance degradation under unexpected topic shifts. Although the paper includes real-team evaluations, these do not establish how frequently offline rehearsal successfully anticipates high-value information needs in naturally evolving scientific discussions.

W2.3. The experimental comparisons do not isolate the contribution of rehearsal. The baselines primarily perform retrieval and reasoning online, whereas DTR benefits from substantial offline precomputation. Consequently, the latency advantage largely follows from the different allocation of computation. A critical missing baseline is a compute-matched offline preparation method that directly performs deep research using the meeting topic, proposal, and available team context, without simulating multi-party meetings, followed by the same online retrieval pipeline. Without such a comparison, it remains unclear whether meeting rehearsal provides meaningful benefits beyond conventional advance knowledge preparation. Additionally, the human evaluation compares DTR using Qwen-Plus against GPT-5.2-Thinking, rather than providing a backbone-controlled comparison, further limiting causal attribution of the reported quality improvements.

Questions for authors

Q1. Regarding the fundamental predictability assumption:

What empirical evidence supports the claim that future scientific discussion trajectories and their important information needs can be meaningfully anticipated from meeting topics and historical context?

On STMD-Real, what proportion of actual information-seeking turns can be adequately addressed using the precomputed rehearsal knowledge? How does response quality and latency change as discussions diverge from anticipated topics, and how frequently is fallback required?

Q2. Regarding benchmark validity and potential information leakage:

How do the authors ensure that the Prophet LLM's access to the full paper and golden conclusion does not artificially guide synthetic meetings toward predetermined outcomes?

Can the authors evaluate DTR on meeting trajectories generated without access to golden conclusions or future results, and examine whether its advantages persist under less predictable discussion settings?

Q3. Regarding the necessity of meeting rehearsal:

Can the authors compare DTR against a compute-matched offline deep research baseline that prepares a knowledge database directly from the meeting topic, project proposal, and team context, without multi-agent meeting simulation?

Under matched offline budgets, retrieval settings, and response backbones, how much additional improvement can be attributed specifically to rehearsal rather than to offline computation and knowledge caching?

Q4. Additional Notes for Problem Framing

Given that the main quantitative evaluation relies on LLM-simulated meetings, would it be more appropriate to position DTR as a framework for multi-agent scientific collaboration rather than real-time human–AI collaboration?

Agent-generated discussions may offer a more controlled and predictable setting for offline rehearsal, whereas human meetings can evolve unpredictably due to unexpected findings, disagreements, and emerging ideas.

If the authors intend to retain the human–AI collaboration framing, stronger empirical evidence from naturally occurring human meetings would be needed to justify the applicability of rehearsal to this setting.

Reviewer_xnmFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Deep Think with Rehearsal (DTR) to reduce response latency in AI-assisted scientific team meetings. Before a meeting, DTR simulates multi-party discussions conditioned on the meeting topic, historical team memory, and participant personas. It applies confidence-based filtering and selectively invokes tool-augmented reasoning to construct a rehearsal knowledge database. During the live meeting, the response model retrieves relevant notes to generate answers, reducing the need for online search and extensive reasoning. The paper also introduces the Scientific Team Meeting Dataset (STMD), combining PubMed-grounded synthetic discussions with meeting transcripts from three biomedical research teams. Experiments report lower online latency and improvements in most automatic response-quality metrics, with supplementary evidence from component ablations, rehearsal-budget analysis, and expert evaluation.

Critical strengths

1 The problem is practically relevant and the framework is well aligned with its objective Balancing response quality and latency is a practical challenge for interactive AI assistants. For scientific meetings with predefined topics and available background information, moving expensive retrieval and reasoning into a preparation phase is a reasonable approach with practical potential. DTR clearly separates offline preparation from online response generation, providing a feasible way to reduce waiting during live interaction. 2 Online improvements are reasonably consistent across the tested configurations Across three model backbones and two synthetic benchmarks, DTR reduces first-token latency to approximately 0.9-1.3 seconds while improving most automatic quality metrics. The paper also reports offline preparation costs and examines individual components, rehearsal budgets, and alternative allocations of online and offline computation. These experiments provide initial evidence about the sources of the gains and the computational trade-offs. 3 The dataset and expert involvement add value STMD combines scalable synthetic discussions with real laboratory meeting transcripts, providing a useful resource for studying AI assistance in scientific collaboration. Researchers evaluate both dataset quality and model responses in terms of quality, helpfulness, and personalization. These assessments complement the automatic metrics and provide some evidence of practical usefulness.

Critical weaknesses

1 A crucial baseline is missing leaving the specific contribution of multi-party rehearsal unclear The paper should include a more direct baseline that retrieves relevant literature and summarizes information from the meeting topic and historical records to construct a RAG knowledge base under the same offline computation and retrieval budget, while controlling the backbone and available information. The main comparisons currently contrast DTR with extensive advance preparation against methods that perform retrieval and reasoning online. Consequently, the results do not distinguish gains from multi-party rehearsal and its associated mechanisms from gains due to acquiring and organizing more relevant material in advance. The comparison between online and offline rehearsal in Table 3 supports moving computation before the interaction, but does not answer whether this rehearsal mechanism is necessary to construct an effective knowledge base for the meeting. This also affects the novelty assessment. The cited PRINCIPLES work already uses offline simulations, reusable memory, and online retrieval, and analyzes inference efficiency. Comparisons with simpler offline preparation strategies and appropriately adapted simulation-memory methods are therefore needed to isolate the additional benefits of team-conditioned rehearsal, scientific evidence acquisition, and confidence-based filtering. Kim et al. (2025), PRINCIPLES. 2 Automatic evaluation does not adequately support the claim of improved scientific reasoning quality ROUGE and BLEU primarily measure lexical overlap and cannot adequately assess scientific correctness, reasoning validity, or the value of suggestions. Since the reference answers are derived from the source papers' existing conclusions, higher scores do not necessarily indicate deeper or more useful scientific advice. Expert evaluation provides some complementary evidence, but factual accuracy, evidential support, and actionability still require more direct assessment. 3 The human evaluation protocol and uncertainty reporting are insufficiently detailed The paper reports average ratings over 200 feedback instances from three teams, but does not adequately explain how these instances are distributed across participants, teams, and meetings, or whether responses are evaluated through matched comparisons on the same questions. Multiple ratings from the same participant or team should not automatically be treated as independent observations; the handling of repeated ratings and within-team correlations needs to be clarified. Table 2 also provides no confidence intervals or team-level breakdowns. It is therefore difficult to determine whether the average advantages are consistent across teams and meetings or driven mainly by a subset of the evaluation instances. 4 The assumed meeting setting and its predictability remain insufficiently validated DTR relies on several conditions holding together: sufficient historical meeting records, relatively stable team membership, advance knowledge of the agenda, and substantial coverage of actual information needs through offline simulation. The paper does not adequately establish how representative this combination is of the scientific collaboration settings it addresses. In particular, knowing a meeting's topic does not necessarily make its specific questions, key evidence, or required reasoning predictable. For newly assembled interdisciplinary teams, the topic may be known while the historical data needed to construct team memory, participant personas, and team-specific confidence thresholds are unavailable. Conversely, recurring project meetings may provide extensive history but be driven by newly obtained results, unexpected experimental failures, or changing project constraints. The central needs may then involve interpreting new evidence, diagnosing experimental problems, and revising research plans using information that did not exist during rehearsal. The paper has not sufficiently demonstrated that the prepared knowledge and simulated trajectories can support these tasks effectively. Moreover, the synthetic historical and evaluation meetings are generated from the same completed paper and guided by predefined topic sequences. This may overestimate the predictability of real meetings and amplify the apparent advantages of rehearsal. The three-team evaluation does not directly establish rehearsal coverage or performance when new information emerges or the discussion changes direction. The authors should define the intended meeting category more precisely, distinguish tasks such as literature consultation, project-progress discussion, and interpretation of new results, and report both rehearsal coverage and performance beyond that coverage to support broader claims about scientific collaboration.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.