acceptodds
Under review as a conference paper at ICLR 2027

Asynchronous Reasoning: Training-Free Interactive Thinking LLMs

Abstract

Many state-of-the-art LLMs are trained to think before giving their answer. Reasoning can greatly improve language model capabilities, but it also makes them less interactive: given a new input, a model must stop thinking before it can respond. Real-world use cases such as voice-based or embodied assistants require an LLM agent to respond and adapt to additional information in real time, which is incompatible with sequential interactions. In contrast, humans can listen, think, and act asynchronously: we begin thinking about the problem while reading it and continue thinking while formulating the answer. In this work, we augment LLMs capable of reasoning to operate in a similar way without additional training. Our method uses the properties of positional embeddings to enable LLMs built for sequential generation to simultaneously think, listen, and write outputs. We evaluate our approach on math, commonsense, and safety reasoning: it allows models to generate accurate thinking-augmented answers while reducing time to first non-thinking token from minutes to 5s and the overall delays by up to .

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Reasoning, Inference, Asynchronous

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CmKbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AsyncReasoning, a training-free method that enables large language models to reason and generate responses concurrently. The method uses positional embeddings and a shared KV cache to support asynchronous thinking and writing. It also introduces a mode-switching mechanism that determines when the model should continue responding or pause for further reasoning. The authors evaluate their approach on mathematical reasoning, commonsense reasoning, safety, and interactive tasks. Experimental results show that AsyncReasoning significantly reduces response latency while preserving most of the accuracy benefits of reasoning models.

Critical strengths
  • The idea of allowing models to think and generate responses concurrently without additional training is interesting. The use of positional embeddings and a shared KV cache provides a practical solution to this problem.

  • The experiments cover multiple models and reasoning benchmarks, and the results demonstrate significant improvements in response latency while maintaining relatively high accuracy.

Critical weaknesses
  • The proposed method still leads to some accuracy degradation compared with standard reasoning. For example, Table 1 shows that the accuracy of GPT-OSS-120B on AIME-2025 decreases from 0.77 to 0.66 when using AsyncReasoning. Although the latency improvement is significant, it would be helpful to provide further analysis of why this accuracy loss occurs and whether it can be reduced.

  • The evaluation of real-world interactivity could be more comprehensive. Although the paper includes experiments with additional user inputs, most evaluations are based on mathematical and reasoning benchmarks. It would be interesting to see how the method performs in more realistic scenarios, such as continuous conversations or frequent user interruptions.

Questions for authors

please refer to "Weaknesses"

1 comment

Thank you for your review and suggestions! We address your concerns below.

(Weakness 1) The proposed method still leads to some accuracy degradation compared with standard reasoning. For example, Table 1 shows that the accuracy of GPT-OSS-120B on AIME-2025 decreases from 0.77 to 0.66 when using AsyncReasoning. Although the latency improvement is significant, it would be helpful to provide further analysis of why this accuracy loss occurs and whether it can be reduced.

We report our error analysis in Appendix C for the main tasks and further analyze safety failure modes in appendix L. In the setup you described, OPT-OSS-120B scored 77% with medium-effort sequential reasoning, 66% with AsyncReasoning, and 51% with low-effort reasoning (Table 16). The main cause is that the Writer commits to an answer before the Thinker has verified it. We observed similar behavior on Qwen3-0.6B/MATH-500 (Appendix C). In our ablation analysis (Section 4.1, Appendix F), we found that this can be mitigated by tuning the prompts and mode switching mechanisms for the specific task. However, to keep our method simple and avoid experimental confounders, we use the same prompt and mechanism across all models and tasks. We report accuracy drawdowns where they occur and, overall, we find that AsyncReasoning offers a valuable trade-off between accuracy and response speed.

The evaluation of real-world interactivity could be more comprehensive. Although the paper includes experiments with additional user inputs, most evaluations are based on mathematical and reasoning benchmarks. It would be interesting to see how the method performs in more realistic scenarios, such as continuous conversations or frequent user interruptions.

We agree wholeheartedly! The main problem with real-world reasoning interactivity is that it is difficult to measure with scientific rigour, as it depends on many subtle cues in human interaction, which is why we evaluated more specific scenarios that can be measured.

For non-mathematical reasoning, GPQA Diamond measures reasoning outside mathematics, including biology and chemistry problems. In turn, ZebraLogic tests logical deductive reasoning without equations.

For more frequent user interruptions, while our MATH-500 Sharded evaluations only have a single interruption, the Lost in Conversation math problems (Figure 6) include 5-6 corrections per task. We agree that certain real-world scenarios can have even more dynamic interruptions that depend on the agent’s responses to previous user feedback, but these are notoriously difficult to capture in a benchmark.

A user study would indeed be interesting, even though it introduces complications. Since each user has a different speech cadence and habits on how soon they ‘interrupt’ the assistant, the results will vary significantly between populations, especially for dynamic conversations. To that end, we measure full end-to-end delay on a subset of 50 MATH-500 samples on Qwen3-30B-A3B-2507-Thinking in the same setup as in Section 4.2.

The main difference from our previous evaluations is that we do not simulate user via the TTS but actually spoke to the AsyncReasoning agent running on a single H100 GPU (including Whisper-base ASR and Tortoise TTS). The results below are the wall time from the moment the user stops talking to the moment the system begins speaking, and the frame-by-frame silence duration. Compared to our main evaluations, this includes both automated speech recognition and software / hardware delays.

Q3-30B-A3B-2507TTTFT (s.)Total Delay (s.)
AsyncReasoning (end-to-end)4.6439.887
AsyncReasoning (Section 4.2)3.9019.145
Baseline w/ think (end-to-end)895.599895.599
Baseline w/ (Section 4.2)894.884894.884

The new end-to-end are slightly (<1s) higher than the controlled evaluation from Section 4.2 due to speech recognition, input hardware and software delays. Note that the difference in total delays in our main results is much larger (tens to hundreds of seconds). We hope that our answers help alleviate your concerns. If you have another specific scenario you would like to test, please suggest it in a follow-up response.

Reviewer_5L5LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method that allows thinking and answering streams to progress independently

Context is built as three separate KV cache blocks for input, thinking and answer. New tokens are added to each block independently by the environment or thinking and answering generators.

Thinking and answering generators see an approximate "full context" build from combining those separate input, thinking and output KV-caches. Since tokens can be appended to the context where any of the blocks finish, one would need to prefill again after each token is inserted/appended to have the exact new context. Instead of this, the approach proposed just recomputes the ROPE rotation for the attention-projection of each hidden vector query. This allows to build this context dynamically, even if tokens are inserted in the middle. This seems to be the same mechanism as in the Hogwild! paper.

Because all block context grow at the same time, it also introduces a control that can stop the answer stream from writing. This is implemented as binary question every 20 tokens. The write also pauses at every paragraph until the answering control tells it to resume.

Critical strengths
  • Its a simple traning-free method applicable to any thinking model

  • The experiments use a diverse set of models (Qwen3 0.6B-235B, GPT-OSS) showing positive results

Critical weaknesses
  • The statement "the only thing that makes [models] aware of sequence order is positional encoding" is not true. Causal masking makes any representation beyond first attention encode position implicitly which precludes true translation of blocks.

  • The above is relevant for a general weakness of the aproach: just rotating the concatenated KV-cache attention projectcions will only match the true prefill in the input embedding. All hidden representations of the layers will not match the true prefil recomputation. I would expecty this to limite the efficiecny of the approach.

  • I could have missed something, but the code provided in the supplementary material does not match the paper description? In the code the thinker never sees the write. async_reasoning/cache.py Line 64 has

thinker_view = (self.input_prompt, self.input_block, self.thinker_output)

this seems to contradict the paper.

  • The Hogwild! paper cited as inspiration seems to share a lot of the aspects of this work. A more detailed explanation of where this paper innovates over Hogwild! is necessary.

  • Other works missing that seem related

STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models Interleaved Reasoning for Large Language Models via Reinforcement Learning

  • Presentation issues: It was hard to understand how the system worked and Figure 2 font is too small.
Questions for authors
  • Can you explain the insconsistency between paper and code (if any)?
  • Please elaborate here on what this paper adds in relation to Hogwild!
2 comments

Thank you for your feedback! Your review raises several suggestions that can improve our presentation. We updated our PDF to incorporate your improvements.

(Weakness 1) The statement "the only thing that makes [models] aware of sequence order is positional encoding" is not true. Causal masking makes any representation beyond first attention encode position implicitly which precludes true translation of blocks.

Thank you for catching that! In the context of that sentence, we were analyzing transformers in general and needed to establish that 1) MLPs, embeddings, and attention projections are order-equivariant, and 2) in the general transformer encoder, positional embeddings are the only thing that informs the model about the sequence order. You are right that, in a transformer decoder, the model can implicitly encode position because of the causal attention mask, and we updated the paper to reflect that.

(Weakness 2) The above is relevant for a general weakness of the aproach: just rotating the concatenated KV-cache attention projectcions will only match the true prefill in the input embedding. All hidden representations of the layers will not match the true prefil recomputation. I would expecty this to limite the efficiecny of the approach.

You are right that AsyncReasoning with concurrent thinking and writing is not mathematically equivalent to sequential reasoning. If the Thinker adds a thought after the Writer finishes a paragraph, the cached paragraph still reflects what was available when it was written. We would like to note that we never claim AsyncReasoning to be mathematically equivalent: instead, we evaluated 10 models to verify its practical efficiency across 6 different reasoning benchmarks and found that it can provide faster responses with comparable accuracy.

(Weakness 3 / Question 1) Can you explain the insconsistency between paper and code (if any)?

In Section 4, the thinker does not attend to the writer as it thinks, but it does attend to the writer when deciding whether to pause it (as depicted in Figure 1). We found that letting the thinker see the writer while generating thoughts did not meaningfully improve performance, so we only allowed it for mode-switching, as we explain in Appendix B. We agree that this could have been explained better and will clarify this in Section 3.1.

(Weakness 4 / Question 2) What does this paper add in relation to Hogwild!?

Our main additions stem from solving a different problem. In Hogwild!, two or more symmetric workers constantly reason on the same task until they find the answer. In contrast, our workers are asymmetric and they do not simply generate tokens in parallel. We implement an adaptive mode switching mechanism that lets the thinker control when the writer needs to pause, giving the LLM control over the thinker-writer asynchrony. In Appendices C and F, we show that the design of this mechanism is crucial to maintain high accuracy for asynchronous reasoning. On a technical side, we modify concurrent attention runtime to support dynamic mode switching and support for GPT-OSS models. That said, we build our inference on top of Hogwild!’s shared KV cache and RoPE adjustment mechanisms, as we state in Section 3.3 (L208-209) and later in the implementation details.

(Weakness 5) Other works missing that seem related. STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models Interleaved Reasoning for Large Language Models via Reinforcement Learning

Thank you for suggesting additional related work! STITCH alternates reasoning and speech chunks, using playback time to compute the next reasoning chunk. They train a model with reinforcement learning to interleave reasoning and answering. Our method does not require training and can advance both streams in one forward pass. In Section 4.1, we compare AsyncReasoning against training-free interleaved reasoning based on a different paper, but we cited STITCH as an additional relevant work.

We agree that Figure 2 text is a little hard to read: we had to shrink it for spacing reasons. If accepted, we will use the additional content page to improve presentation and we will discuss STITCH in more detail in the updated paper. If you have any other suggestions, we would be glad to continue our discussion below.

I thank the authors for their clarifications and their intention to improve manuscript clarity. My concerns about the novelty compared to Hogwild! stays. I will keep my score.

Reviewer_XXKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AsyncReasoning, an inference-time method that allows a reasoning model to generate private thoughts and a public response concurrently, without additional training. It reuses a shared KV cache with view-specific positional adjustments and periodically asks the model whether the response stream should continue or wait for further reasoning. Experiments cover mathematical reasoning, knowledge and logic benchmarks, safety-oriented prompting, and inputs arriving during generation. The reported results suggest that some models can retain much of the benefit of sequential reasoning while substantially reducing time to the first public token and cumulative silence during speech playback.

Critical strengths
  1. The paper addresses a useful problem: long reasoning traces can make otherwise capable models difficult to use interactively. Adapting shared-cache concurrency to private reasoning and public response generation is a practical direction, particularly because it does not require retraining the underlying model. The ability to pause the response stream also gives the method more flexibility than simply forcing the model to answer early.

  2. Some results show a promising accuracy–latency trade-off. In Table 13, Qwen3-32B retains approximately 0.94 accuracy on MATH-500 while TTFT decreases from 649.47 to 4.32 seconds and cumulative silence decreases from 649.94 to 102.10 seconds. The appendices also examine switching frequency, quantization, batching, and switching failures. These analyses provide useful information about when the approach works and where coordination becomes difficult.

Critical weaknesses
  1. Several inconsistencies prevent me from establishing what the main results support. On ZebraLogic, Table 1 reports thinking/async accuracies of 0.71/0.68 for Qwen3-32B, whereas Table 17 reports 0.85/0.66. The implied accuracy loss is therefore either 3 or 19 percentage points, which materially changes the assessment. There also appears to be a unit or column mix-up: the async AIME TTFT values for GPT-OSS-20B and GPT-OSS-120B are 24.15 and 24.63 seconds in Table 1, but 2.83 and 9.32 seconds in Table 16. The former numbers exactly match the “Steps to 1st token” column in Table 16.

    The default algorithm is also unclear. Section 3.1 and Figures 2–3 show the Thinker attending to the Writer, but Appendix B defines thinker_view = [input prompt, thinker output]. Appendix F then describes allowing the Thinker to see the Writer as a separate variant. This leaves an important question unresolved: do the main experiments evaluate bidirectional interaction, or a Thinker that does not read the Writer, with a separate control query monitoring both? These discrepancies affect both the interpretation of the results and reproducibility.

  2. The latency evaluation does not isolate the effect of scheduling from that of longer public responses. Cumulative silence is relevant to a voice interface, but it depends on how much speech the system produces. In Table 16, the average public response on AIME grows from about 573 to 13,587 tokens for Qwen3-30B-A3B, and from about 413 to 7,377 tokens for Qwen3-235B-A22B. Longer playback can cover more of the remaining reasoning time. The current measurements therefore leave open how much of the improvement comes from better scheduling and how much comes from speaking for longer.

    Appendix E provides useful end-to-end measurements, but these still focus on the start of speech and cumulative silence. I would also need the time to the final answer and total playback duration, ideally with a comparison controlling response length, to judge the practical gain. This matters because the paper’s motivation concerns timely assistance, and the first public token need not contain useful progress toward the answer.

Questions for authors
  1. Which results in Tables 1, 16, and 17 correspond to the final experimental configuration? Please explain the discrepancies, provide a consistent set of results, and state whether the corrections change the conclusions about accuracy retention or latency.

  2. Does the Thinker attend to the Writer in the main experiments? Please specify the actual cache views and update order, and reconcile them with Section 3.1, Figures 2–3, and Appendices B and F.

  3. For a representative model and benchmark, how do AsyncReasoning and sequential reasoning compare in time to the final correct answer and total speech duration? Does the reduction in cumulative silence remain substantial when public response length is controlled?

1 comment

Thank you for your review and feedback! We address your concerns and answer your questions below.

(Weakness 1 / Question 1) Which results in Tables 1, 16, and 17 correspond to the final experimental configuration?

We re-checked these tables against our experiment logs and found that tables 16 and 17 have the correct results. It turned out that, on AIME-2025, Table 1 mistakenly reports decoding steps (24.15 and 24.63) as seconds. Table 16 gives the correct async TTFTs: 2.83 s for GPT-OSS-20B and 9.32 s for GPT-OSS-120B. On ZebraLogic, Table 17 matches the original Qwen3-32B runs. Sequential thinking solved 852/1,000 puzzles and AsyncReasoning solved 656/1,000. Cumulative silence fell from 940.27 to 138.76 s, at a cost of 19.6 percentage points in accuracy. Thank you for spotting that! For consistency, we also double-checked the rest of our results (10 models and 6 benchmarks) to verify they are correct. We will fix the typos in Table 1.

(Weakness 1 / Question 2) Does the Thinker attend to the Writer in the main experiments?

In Section 4, the thinker does not attend to the writer as it thinks, but it does attend to the writer when deciding whether to pause it (Figure 1). We found that letting the thinker see the writer while generating thoughts did not meaningfully improve performance, so we only allowed it for mode-switching, as we explain in Appendix B. We agree that this could have been explained better and will clarify this in Section 3.1.

(Weakness 2 / Question 3) Does silence fall at comparable response lengths? When is the correct answer heard?

On 1,402 SpokenMQA questions, Qwen3-4B averaged 311 vs. 312 public tokens with sequential and async reasoning. Cumulative silence fell from 227.16 to 0.84 s, with accuracy of 81.5% vs. 81.0% (Table 18). These lengths are comparable on average. We did not match them per question. On Qwen3-32B/MATH-500, async playback ended after 203 s on average, including 101 s of speech and 102 s of silence. Sequential thinking took 649 s to produce its first public token. We did not measure when the correct answer first became audible (Tables 6 and 13). We hope that these answers help alleviate your concerns. Please tell us if you have any follow-up questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.