acceptodds
Under review as a conference paper at ICLR 2027

Asynchronous Large-to-Small Handoff for Efficient Reasoning

Abstract

Long chain-of-thought (CoT) reasoning improves the performance of large language models on complex tasks, but also incurs substantial inference cost and latency. We investigate whether a smaller model can help complete an ongoing large-model reasoning trajectory faster while preserving the larger model's answer quality. We propose , a framework that launches small-model continuations from intermediate large-model reasoning states, creating an online race between continued large-model reasoning and candidate small-model completions. We further train a prefix-conditioned Reasoning Finisher with reinforcement learning to improve the accuracy–latency trade-off. Across ten benchmarks spanning mathematics, science, and code generation, ALSH reduces latency by compared with standalone large-model inference in our main Qwen3.5-35B/Qwen3.5-2B setting, with a relative improvement in dataset-macro accuracy, demonstrating a stronger accuracy–latency balance than prior large–small collaboration methods. Experiments across different model scales and families further support the generality of ALSH.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Chain-of-Thought Reasoning, Inference Acceleration, Large-Small Model Collaboration, Asynchronous Reasoning Handoff, Reinforcement Learning, Reasoning Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_WLEuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Asynchronous Large-to-Small Handoff (ALSH),a framework for improving the efficiency of LLM reasoning through collaboration between large and small models. The main idea is to let the large model continue reasoning while asynchronously launching small-model completion attempts from intermediate reasoning prefixes.The system returns the first eligible completion, and further train a prefix-conditioned Reasoning Finisher using rl to improve the effectiveness of these handoffs. Experiments across ten benchmarks covering mathematics, scientific reasoning, and code generation report improvements in both accuracy and latency compared with standalone large-model inference.

Critical strengths
  1. The paper includes extensive experiments across multiple benchmarks, model scales, and model families, demonstrating the potential generalizability of the approach.

  2. The authors provide several ablation studies and system-level measurements to investigate the effects of finisher training, handoff configurations, and inference efficiency.

Critical weaknesses

1. Unclear practical motivation for the reasoning handoff setting.

I am not fully convinced of the practical motivation for the proposed large-to-small reasoning handoff setting. It is unclear what real-world scenarios would benefit from such a specialized mechanism, particularly when both models are deployed on the server side. Why not directly optimize the inference efficiency of the large model, or use model routing to select an appropriate model at the beginning of each request? More established approaches, such as adaptive early stopping, model routing, and speculative decoding, already provide alternative ways to improve inference efficiency. Although these approaches operate at different granularities, I believe comparisons with representative methods would be important to establish the practical necessity and advantages of the proposed setting.

2. Significant redundant computation and questionable overall efficiency.

The asynchronous design introduces substantial redundant computation, since the large model continues generating while multiple small-model probes run concurrently, and most of these computations may eventually be discarded. Although the authors report a 52.7% latency reduction in the main setting, the total computational cost actually increases by 25.2%. This raises concerns about whether the method truly improves overall inference efficiency or simply trades additional computational resources for lower response latency. While the paper reports analytical FLOPs, it lacks a comprehensive evaluation of actual serving costs, GPU utilization, throughput, and performance under fixed hardware budgets. Without such analysis, the practical efficiency benefits remain unclear.

3. Unpolished and apparently AI-generated presentation.

The paper gives a strong impression of being hastily assembled with AI assistance. In particular, Figure 1 has an unmistakably AI-generated appearance, with a generic visual style that lacks the precision and professionalism expected from a technical research paper. More importantly, the figure does not clearly communicate the technical details of the proposed asynchronous handoff mechanism, such as concurrent execution, scheduling, and handoff commitment. I find this presentation rather rough and insufficiently polished for an ICLR submission.

4. Potentially unfair comparisons with existing collaborative reasoning methods?

The experimental comparison does not sufficiently isolate the contributions of the asynchronous handoff mechanism and the specially trained Reasoning Finisher. In particular, Tandem and RelayGen use the original, untrained Qwen 2B model, whereas the strongest ALSH configuration uses a prefix-conditioned model trained with reinforcement learning. Furthermore, the Tandem implementation effectively forces a handoff after 1K tokens, resulting in a very aggressive accuracy-latency trade-off. Consequently, the reported improvements may largely come from the stronger small model or the particular baseline configurations rather than the proposed asynchronous scheduling mechanism. A more convincing evaluation would compare different handoff strategies using the same trained finisher and report their accuracy-latency trade-off curves.

Questions for authors
  1. Practical motivation: Couldthe authors clarify the real-world deployment scenarios where asynchronous reasoning handoff would be preferable to simpler approaches such as model routing, adaptive early stopping, or speculative decoding? What are the unique practical advantages of this setting?

  2. Computational efficiency: Given that the main configuration increases total FLOPs by 25.2%, how does ALSH perform in terms of actual GPU costs, throughput, and latency under a fixed hardware budget? Is the additional computational overhead justified in realistic serving scenarios?

  3. Baseline fairness: Could the authors compare ALSH against existing handoff methods using the same trained Reasoning Finisher? In particular, how would Tandem perform with properly tuned handoff thresholds rather than an effectively fixed 1K-token handoff? A comparison of accuracy-latency trade-offs under matched conditions would behelpful.

5 comments
AuthorsOpenReview ↗

Motivation and alternatives

Thank you for this question. Below we explain how ALSH differs from the approaches you mentioned.

Weakness.

1. Unclear practical motivation for the reasoning handoff setting.

I am not fully convinced of the practical motivation for the proposed large-to-small reasoning handoff setting. It is unclear what real-world scenarios would benefit from such a specialized mechanism, particularly when both models are deployed on the server side. Why not directly optimize the inference efficiency of the large model, or use model routing to select an appropriate model at the beginning of each request? More established approaches, such as adaptive early stopping, model routing, and speculative decoding, already provide alternative ways to improve inference efficiency. Although these approaches operate at different granularities, I believe comparisons with representative methods would be important to establish the practical necessity and advantages of the proposed setting.

Question 1.

Practical motivation: Couldthe authors clarify the real-world deployment scenarios where asynchronous reasoning handoff would be preferable to simpler approaches such as model routing, adaptive early stopping, or speculative decoding? What are the unique practical advantages of this setting?


Response

A small model that cannot solve a question from scratch may still finish it from a large model's reasoning prefix. As discussed in Section 2, prior works like R2R, Hawkeye, Tandem and RelayGen have demonstrated efficiency gains from large–small collaboration within a request. Our work focuses on when to hand off reasoning and how to help the small model finish efficiently. We combine asynchronous handoff attempts with a finisher trained to complete partial reasoning prefixes.

Relative to request-level routing. At native 256k context, the same trained 2B model solves 40/200 OlymMATH questions alone (20.0%), whereas ALSH solves 149/200 (74.5%), with all 200 outputs committed by the finisher. The inherited reasoning creates completion opportunities unavailable to routing solely from the original question.

Relative to adaptive early stopping. Adaptive early stopping decides when the current model should stop reasoning and return an answer. ALSH instead lets a smaller model continue the remaining reasoning from the large model's intermediate state at a lower computational cost per token. Following your suggestion, we will include DEER as an adaptive early stopping baseline and report the results in the revised manuscript.

Relative to speculative decoding. Speculative decoding is orthogonal to ALSH. It speeds up token generation for a target model, while ALSH decides when to hand the remaining reasoning to a smaller model. It could therefore accelerate either the large model or the finisher in ALSH. We are also evaluating how much additional latency reduction can be achieved by combining ALSH with speculative decoding.

For example, adding DFlash to ALSH's large model further reduces mean latency from 60.73 to 41.59 s on AIME 2025. We will report the complete ten-benchmark comparisons in the revised manuscript.

AuthorsOpenReview ↗

Revised Figure 1

Weakness 3.

More importantly, the figure does not clearly communicate the technical details of the proposed asynchronous handoff mechanism, such as concurrent execution, scheduling, and handoff commitment.

Response

We have redrawn Figure 1 and revised its caption in the manuscript. The updated figure explicitly shows concurrent execution, SCP launches at successive anchors, the TEW check, and both commitment outcomes.

AuthorsOpenReview ↗

The same trainer comparison

Thank you for this question. Below, we discuss the comparison of handoff methods using the same trained finisher.

Weakness.

4. Potentially unfair comparisons with existing collaborative reasoning methods?

The experimental comparison does not sufficiently isolate the contributions of the asynchronous handoff mechanism and the specially trained Reasoning Finisher. In particular, Tandem and RelayGen use the original, untrained Qwen 2B model, whereas the strongest ALSH configuration uses a prefix-conditioned model trained with reinforcement learning. Furthermore, the Tandem implementation effectively forces a handoff after 1K tokens, resulting in a very aggressive accuracy-latency trade-off. Consequently, the reported improvements may largely come from the stronger small model or the particular baseline configurations rather than the proposed asynchronous scheduling mechanism. A more convincing evaluation would compare different handoff strategies using the same trained finisher and report their accuracy-latency trade-off curves.

Question 3.

Baseline fairness: Could the authors compare ALSH against existing handoff methods using the same trained Reasoning Finisher? In particular, how would Tandem perform with properly tuned handoff thresholds rather than an effectively fixed 1K-token handoff? A comparison of accuracy-latency trade-offs under matched conditions would behelpful.


Response

The gains do not come solely from finisher training. As reported in the original manuscript, ALSH with the original, untrained 2B model already improves accuracy and latency over standalone in our main setting, and finisher training further improves both (Table 1).

Table 1. Finisher training in the main setting (macro averages)

Method / finisherAccuracy (%) ↑Latency (s) ↓
Large model standalone72.92175.31
ALSH, original 2B74.15169.34
ALSH, question-only trained79.19123.57
ALSH, prefix-trained81.6282.91

Tandem threshold calibration. The effective 1K handoff in our original comparison was selected through held-out calibration using Tandem's published guidance budgets. Following your suggestion, we have recalibrated the handoff thresholds on a larger calibration set of 512 questions using the same trained finisher as ALSH. For the 100/500/1,000-token guidance stages, the recalibrated thresholds are (0.95, 0.95, 0.0), close to the original (1.0, 1.0, 0.0). The comparison below uses these recalibrated thresholds. We will also test additional handoff thresholds.

Comparison with the same trained finisher. We agree that using the same trained finisher provides a fairer comparison of handoff policies. As requested, we have evaluated ALSH, Tandem and RelayGen with the same Qwen 35B-A3B backbone and trained finisher checkpoint. Table 2 reports macro-average accuracy and latency across all ten benchmarks under our main settings. We will report the per-benchmark results in the revised manuscript.

Table 2. Macro-average results across ten benchmarks

MethodAccuracy (%) ↑Latency (s) ↓
Standalone72.92175.31
Tandem57.8145.84
RelayGen73.40190.74
ALSH81.6282.91

Accuracy–latency comparison across settings. The accuracy–latency plot will show ALSH and Tandem under different settings alongside RelayGen and the other baselines. We will include this plot in the revised manuscript.

Reviewer_ga5HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Asynchronous Large-to-Small Reasoning Handoff (ALSH), a framework that launches multiple small-model “completion probes” from intermediate large-model reasoning prefixes while allowing the large model to continue uninterrupted; the first eligible completion (small or large) is committed. A prefix-conditioned “Reasoning Finisher” is trained with reinforcement learning to efficiently and correctly complete inherited prefixes within a bounded continuation window. Across 10 reasoning and coding benchmarks, the authors report substantial latency reductions (~52.7%) while improving dataset-macro accuracy by 11.9% over standalone large-model inference in their main Qwen 35B→2B configuration, with additional evidence of cross-scale and cross-family generalization.

Critical strengths

Novelty

  • Introduces the idea of decoupling handoff attempts from commitment: small-model probes are launched asynchronously without interrupting the ongoing large-model trajectory, eliminating the one-shot handoff decision bottleneck common in prior work.
  • Formalizes a Takeover Eligibility Window (TEW) to bound small-model continuation and operationalize takeover eligibility.
  • Trains a prefix-conditioned “Reasoning Finisher” with RL (GRPO) on leakage-free large-model prefixes, encouraging short, correct completions that align with the TEW constraint.

Experimental rigor

  • Evaluates on 10 diverse benchmarks (math, science, code) with 3,595 examples and reports both accuracy and live serving latency (including queuing/scheduling).
  • Includes matched controls that disentangle contributions from the asynchronous framework and finisher training (original 2B, question-only RL, prefix-conditioned RL; window/interval sweeps; LRP construction variants).
  • Demonstrates cross-scale (Qwen 27B/35B/397B) and cross-family transfer without retraining the finisher in several settings.

Clarity

  • Clear problem statement and motivation; intuitive figure and precise formulation of anchors, probes, TEW, and commit policy.
  • Well-structured ablations (conditioning, Wtrain, Wtest, probe interval) and discussion around design choices.

Significance

  • Tackles a practically important challenge: reducing reasoning latency and cost without sacrificing answer quality.
  • The asynchronous race design provides a compelling alternative to one-shot or tightly coupled switch policies and may influence future collaborative inference systems.
Critical weaknesses

Technical limitations or concerns

  • The commit policy appears to rely on structural validity and finishing within the TEW, not correctness. Without a verifier, committing the “first eligible” small-model answer could lock in incorrect outputs that the large model might have ultimately corrected.
  • Compute/energy trade-offs are underexplored: in at least one main setting (Qwen 35B -> 2B), analytical FLOPs increase (+25.2%) despite lower latency, indicating that ALSH can reduce latency at the expense of higher total compute.
  • The system-level practicality of running an advancing large model plus multiple asynchronous small-model probes on shared hardware is only briefly discussed; the effects on throughput, memory/KV-cache pressure, and scheduling fairness need more detail.

Experimental gaps or methodological issues

  • Baseline coverage is limited for the stated design space: comparisons are provided to Tandem and RelayGen, but not to several strong, recent within-trajectory collaboration or early-exit/verification methods (e.g., R2R/RelayLLM/Tri-gReason, DEER/REFRAIN/ThoughtTerminator/CoRE-Eval, speculative reasoning/step drafting). The absence of First-Finish Search (FFS) as a baseline is notable given the shared “race” intuition.
  • No Pareto curves for baselines under multiple budgets; external baselines follow their original settings, which may not reflect their best achievable accuracy–latency trade-offs under the same serving conditions.
  • Statistical uncertainty (CIs/variance) is not reported; given heterogeneous benchmarks and live-serving conditions, uncertainty estimates would strengthen claims.

Clarity issues

  • The definition of “structural validity” across domains (especially for code) is not fully specified in the main text; for code, correctness is execution-based, so “valid structure” may be insufficiently informative to justify early commit.

Missing related work or comparisons

  • While the text cites many relevant methods, empirical comparisons omit several key families (token/span-level relaying, verifier-guided early exit, speculative reasoning of thought steps). Including at least one representative from these families would contextualize the size of ALSH’s gains.
Questions for authors
  1. What fraction of requests commit from the small model versus the large model on each benchmark, and what are the respective conditional accuracies for these two sources?
  2. What scheduling policy and resource isolation (e.g., MIG, multi-process service) are used to avoid starving the large model or other tenants?
  3. What exactly constitutes “structural validity” for TEW eligibility across math, science, and code tasks? For code, is any execution-based check used before commit, or is structural validity purely syntactic?
1 comment
AuthorsOpenReview ↗

Baseline results updated

Hi, we've updated the manuscript.

Following up on our earlier reply about baseline coverage, we have included the early-exit, parallel-search, and speculative-reasoning comparisons you requested in Table 5. The best DEER setting reaches 80.70% at 204.36 s, FFS reaches 79.63% at 155.98 s, and SpecReason reaches 56.35% at 120.21 s. ALSH achieves the highest macro accuracy (81.62%) and the lowest mean latency (82.91 s) among these methods. We also added the accuracy–latency comparison across budgets and settings in Figure 3. The seed stability analysis from our earlier reply is now in Appendix D.3.

Reviewer_mL64OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ALSH, an asynchronous large-to-small reasoning framework that launches small-model continuations from successive large-model prefixes while keeping the large model to continue. The result show it improves dataset-macro accuracy from 72.92% to 81.62% while reducing latency by 52.7%.

Critical strengths

The framework is simple and well motivated.

The main accuracy and latency improvements are substantial. The comparisons with original and question-only-trained finishers provide useful evidence for prefix-conditioned training.

Transfer experiments across model scales and families support the empirical contribution. The paper also reports computational overhead and cases where accuracy decreases.

Critical weaknesses

The efficiency in practical remains incompletely characterized. The main configuration reduces latency but increases total analytical FLOPs by 25.2%.

Uncertainty estimates are missing. Several benchmarks are small, and repeated sampling would help understand the robustness of the reported accuracy improvements.

Questions for authors

na

Reviewer_qa45OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether a small model can finish a large reasoning model's chain of thought earlier without losing the large model's answer quality. ALSH lets the large model keep reasoning and, every 1K tokens, launches a small-model finisher from the current reasoning prefix. It will select whichever finishes faster, the small-model finisher or the large model. To support the system, finisher is trained using RL so it can receive the prefix and finish the request better. Across ten math, science and code benchmarks, the paper reports 52.7% lower latency and accuracy going from 72.92% to 81.62% (Table 1). From what I understand, the main idea is to separate trying a handoff from committing to one, so a failed small-model attempt costs nothing and the large model's trajectory is always kept as a fallback.

Critical strengths
  1. The idea is very interesting. Prior handoff methods make a one-shot decision (Section 2), while ALSH can try many times without disturbing the large model, so in the worst case it falls back to the large model's own answer (Section 3.2). The way it does it asynchronously is a natural choice and it feels close to speculative decoding at the level of reasoning steps. It is also practical to deploy, since it is built on a standard serving stack (vLLM; Appendix C.1).

  2. The finisher-training controls are well designed. The question-only and prefix-conditioned finishers share the same RL objective and budget and differ only in the prefix, and the original 2B is also run with a 16K window so it is not limited by the budget (Section 4.2.1, Table 1). This shows that training on prefixes matters (79.19% to 81.62%, and 123.57 s to 82.91 s).

  3. The efficiency accounting is honest. Latency is measured under live concurrent serving including queueing (Appendix B.1) and the paper reports the 25.2% FLOPs overhead in the main setting instead of hiding it (Section 4.2.1).

Critical weaknesses

Weakness

  1. The accuracy gain may come from the 32K cap on the standalone model rather than from the handoff. Standalone Qwen 35B runs with a 32K generation budget (Appendix D.5), and the paper does not say how runs that hit the cap are scored or how many do. For example, on GSM8K, standalone Qwen 35B reaches only 91.51%, while ALSH with the untrained 2B reaches 95.45% (Table 1). An untrained 2B model is unlikely to out-reason a 35B model on grade-school math, so this gap more likely comes from 35B runs that do not finish within the budget. If so, simpler methods such as forcing the large model to answer at the budget (budget forcing in s1, cited in Section 2) could recover much of the same gain.

  2. The method is limited to tasks with a checkable final answer. A probe is committed only if it stops within the window and passes a structure check such as a boxed answer (Equation 4), and the finisher is trained with a reward that needs a verifiable correct answer (Section 3.3). This fits the evaluated math, multiple-choice and code benchmarks, but it is unclear how ALSH would work on open-ended tasks such as long-form answers or agentic reasoning, and the paper does not discuss this scope.

Formatting and Presentation Issues

  1. The same finisher is called "Prefix-conditioned" in Table 1 and "Leakage-Free LRP" in Table 3, and "Question-only" appears both as a standalone model and inside ALSH in Table 1. Seems like it is the same thing? Why differentiate the naming then?
Questions for authors
  1. How are standalone runs that hit the 32K cap scored, and what fraction of runs hit it on each benchmark? What is the accuracy gap on only the runs that finish within the budget?

  2. Figure 6 already compares ALSH and standalone Qwen 35B with a 256K context. Could the authors report its accuracy and latency as a table for all ten benchmarks? This would show whether the accuracy gain remains when the standalone model is not truncated.

  3. How does ALSH compare with a self-handoff baseline, where the large model itself is forced to answer at each 1K anchor, or with early-exit methods on the large model alone such as Dynasor or DEER (cited in Section 2)? If this gets similar accuracy, does the small model contribute mainly speed?

  4. ALSH uses separate GPUs for the 2B finishers in addition to the large model's GPUs (Appendices C.1 and C.3), and latency includes queueing under up to 16 concurrent requests (Appendix B.1). How many GPUs and finisher replicas does each configuration use, and how does standalone latency change when it is given the same total number of GPUs?

  5. How does ALSH compare in latency with speculative decoding (e.g., EAGLE-3 or DFlash) applied to the same large model, and can the two be combined? Section 2 describes them as complementary, but this is not tested.

3 comments
AuthorsOpenReview ↗

About the structural check and reward

Weakness.

The method is limited to tasks with a checkable final answer. A probe is committed only if it stops within the window and passes a structure check such as a boxed answer (Equation 4), and the finisher is trained with a reward that needs a verifiable correct answer (Section 3.3). This fits the evaluated math, multiple-choice and code benchmarks, but it is unclear how ALSH would work on open-ended tasks such as long-form answers or agentic reasoning, and the paper does not discuss this scope.


Response

The output-format check is a design choice. ALSH does not verify answer correctness at inference time. The shared structural check in Eq. 4 was chosen to cover all ten benchmarks with one task-independent rule. The handoff mechanism does not depend on a boxed-answer format or any specific structural check. For free-form responses, the extra format check can be omitted while retaining natural completion within the window; for agentic tasks, an agent-level completion signal can be used instead.

Training uses no separate structural reward. Our current training uses the task grader's binary correctness reward under a fixed continuation budget. Output-format validity is already included in the grader's overall assessment. The same training framework could therefore be adapted to other tasks using their existing reward signals, such as quality or preference rewards for long-form answers and environment-based task-success rewards for agentic tasks.

AuthorsOpenReview ↗

combination with speculative decoding

Question 5.

How does ALSH compare in latency with speculative decoding (e.g., EAGLE-3 or DFlash) applied to the same large model, and can the two be combined? Section 2 describes them as complementary, but this is not tested.


Response

ALSH can be combined with speculative decoding, as discussed in Related Work (Section 2). Speculative decoding speeds up token generation within a model, so a compatible decoder can accelerate both the large model and finisher while ALSH controls the reasoning handoff.

We evaluated standalone Qwen 35B and ALSH with and without DSpark at the native 256K context. For simplicity, we used the publicly available DSpark weights for Qwen 35B and enabled DSpark only for the large model within ALSH.

Table 1. Macro accuracy and mean latency with and without DSpark

Methodaccuracy (%) ↑latency (s) ↓
Standalone Qwen 35B86.61275.13
Standalone Qwen 35B + DSpark86.09111.51
ALSH83.12113.74
ALSH + DSpark85.5269.76

ALSH alone and standalone Qwen 35B + DSpark achieved similar latency reductions, each cutting mean latency by more than half relative to standalone Qwen 35B, while combining ALSH with DSpark reduced mean latency further, by 37.4% compared with standalone Qwen 35B + DSpark. Thus, reasoning handoff can reduce end-to-end latency even when speculative decoding already accelerates token generation.

AuthorsOpenReview ↗

New results included in the manuscript

Hi, we've updated the manuscript.

Following your suggestion to test budget forcing, we included the 2K answer-reservation control from our earlier reply in Table 1. For your early-exit question, we added DEER to Table 5. The best DEER setting by macro accuracy reaches 80.70% at 204.36 s, while ALSH reaches 81.62% at 82.91 s, improving accuracy by 0.92 percentage points and reducing mean latency by 59.4%. We also included the same-budget and native-256K comparisons from our earlier replies in Appendices D.2 and D.4, along with the full DSpark comparison in Section 4.4 and Appendix D.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.