- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Asynchronous Large-to-Small Handoff (ALSH),a framework for improving the efficiency of LLM reasoning through collaboration between large and small models. The main idea is to let the large model continue reasoning while asynchronously launching small-model completion attempts from intermediate reasoning prefixes.The system returns the first eligible completion, and further train a prefix-conditioned Reasoning Finisher using rl to improve the effectiveness of these handoffs. Experiments across ten benchmarks covering mathematics, scientific reasoning, and code generation report improvements in both accuracy and latency compared with standalone large-model inference.
Critical strengths
-
The paper includes extensive experiments across multiple benchmarks, model scales, and model families, demonstrating the potential generalizability of the approach.
-
The authors provide several ablation studies and system-level measurements to investigate the effects of finisher training, handoff configurations, and inference efficiency.
Critical weaknesses
1. Unclear practical motivation for the reasoning handoff setting.
I am not fully convinced of the practical motivation for the proposed large-to-small reasoning handoff setting. It is unclear what real-world scenarios would benefit from such a specialized mechanism, particularly when both models are deployed on the server side. Why not directly optimize the inference efficiency of the large model, or use model routing to select an appropriate model at the beginning of each request? More established approaches, such as adaptive early stopping, model routing, and speculative decoding, already provide alternative ways to improve inference efficiency. Although these approaches operate at different granularities, I believe comparisons with representative methods would be important to establish the practical necessity and advantages of the proposed setting.
2. Significant redundant computation and questionable overall efficiency.
The asynchronous design introduces substantial redundant computation, since the large model continues generating while multiple small-model probes run concurrently, and most of these computations may eventually be discarded. Although the authors report a 52.7% latency reduction in the main setting, the total computational cost actually increases by 25.2%. This raises concerns about whether the method truly improves overall inference efficiency or simply trades additional computational resources for lower response latency. While the paper reports analytical FLOPs, it lacks a comprehensive evaluation of actual serving costs, GPU utilization, throughput, and performance under fixed hardware budgets. Without such analysis, the practical efficiency benefits remain unclear.
3. Unpolished and apparently AI-generated presentation.
The paper gives a strong impression of being hastily assembled with AI assistance. In particular, Figure 1 has an unmistakably AI-generated appearance, with a generic visual style that lacks the precision and professionalism expected from a technical research paper. More importantly, the figure does not clearly communicate the technical details of the proposed asynchronous handoff mechanism, such as concurrent execution, scheduling, and handoff commitment. I find this presentation rather rough and insufficiently polished for an ICLR submission.
4. Potentially unfair comparisons with existing collaborative reasoning methods?
The experimental comparison does not sufficiently isolate the contributions of the asynchronous handoff mechanism and the specially trained Reasoning Finisher. In particular, Tandem and RelayGen use the original, untrained Qwen 2B model, whereas the strongest ALSH configuration uses a prefix-conditioned model trained with reinforcement learning. Furthermore, the Tandem implementation effectively forces a handoff after 1K tokens, resulting in a very aggressive accuracy-latency trade-off. Consequently, the reported improvements may largely come from the stronger small model or the particular baseline configurations rather than the proposed asynchronous scheduling mechanism. A more convincing evaluation would compare different handoff strategies using the same trained finisher and report their accuracy-latency trade-off curves.
Questions for authors
-
Practical motivation: Couldthe authors clarify the real-world deployment scenarios where asynchronous reasoning handoff would be preferable to simpler approaches such as model routing, adaptive early stopping, or speculative decoding? What are the unique practical advantages of this setting?
-
Computational efficiency: Given that the main configuration increases total FLOPs by 25.2%, how does ALSH perform in terms of actual GPU costs, throughput, and latency under a fixed hardware budget? Is the additional computational overhead justified in realistic serving scenarios?
-
Baseline fairness: Could the authors compare ALSH against existing handoff methods using the same trained Reasoning Finisher? In particular, how would Tandem perform with properly tuned handoff thresholds rather than an effectively fixed 1K-token handoff? A comparison of accuracy-latency trade-offs under matched conditions would behelpful.
5 comments
Revised Figure 1
Weakness 3.
More importantly, the figure does not clearly communicate the technical details of the proposed asynchronous handoff mechanism, such as concurrent execution, scheduling, and handoff commitment.
Response
We have redrawn Figure 1 and revised its caption in the manuscript. The updated figure explicitly shows concurrent execution, SCP launches at successive anchors, the TEW check, and both commitment outcomes.
The same trainer comparison
Thank you for this question. Below, we discuss the comparison of handoff methods using the same trained finisher.
Weakness.
4. Potentially unfair comparisons with existing collaborative reasoning methods?
The experimental comparison does not sufficiently isolate the contributions of the asynchronous handoff mechanism and the specially trained Reasoning Finisher. In particular, Tandem and RelayGen use the original, untrained Qwen 2B model, whereas the strongest ALSH configuration uses a prefix-conditioned model trained with reinforcement learning. Furthermore, the Tandem implementation effectively forces a handoff after 1K tokens, resulting in a very aggressive accuracy-latency trade-off. Consequently, the reported improvements may largely come from the stronger small model or the particular baseline configurations rather than the proposed asynchronous scheduling mechanism. A more convincing evaluation would compare different handoff strategies using the same trained finisher and report their accuracy-latency trade-off curves.
Question 3.
Baseline fairness: Could the authors compare ALSH against existing handoff methods using the same trained Reasoning Finisher? In particular, how would Tandem perform with properly tuned handoff thresholds rather than an effectively fixed 1K-token handoff? A comparison of accuracy-latency trade-offs under matched conditions would behelpful.
Response
The gains do not come solely from finisher training. As reported in the original manuscript, ALSH with the original, untrained 2B model already improves accuracy and latency over standalone in our main setting, and finisher training further improves both (Table 1).
Table 1. Finisher training in the main setting (macro averages)
| Method / finisher | Accuracy (%) ↑ | Latency (s) ↓ |
|---|---|---|
| Large model standalone | 72.92 | 175.31 |
| ALSH, original 2B | 74.15 | 169.34 |
| ALSH, question-only trained | 79.19 | 123.57 |
| ALSH, prefix-trained | 81.62 | 82.91 |
Tandem threshold calibration. The effective 1K handoff in our original comparison was selected through held-out calibration using Tandem's published guidance budgets. Following your suggestion, we have recalibrated the handoff thresholds on a larger calibration set of 512 questions using the same trained finisher as ALSH. For the 100/500/1,000-token guidance stages, the recalibrated thresholds are (0.95, 0.95, 0.0), close to the original (1.0, 1.0, 0.0). The comparison below uses these recalibrated thresholds. We will also test additional handoff thresholds.
Comparison with the same trained finisher. We agree that using the same trained finisher provides a fairer comparison of handoff policies. As requested, we have evaluated ALSH, Tandem and RelayGen with the same Qwen 35B-A3B backbone and trained finisher checkpoint. Table 2 reports macro-average accuracy and latency across all ten benchmarks under our main settings. We will report the per-benchmark results in the revised manuscript.
Table 2. Macro-average results across ten benchmarks
| Method | Accuracy (%) ↑ | Latency (s) ↓ |
|---|---|---|
| Standalone | 72.92 | 175.31 |
| Tandem | 57.81 | 45.84 |
| RelayGen | 73.40 | 190.74 |
| ALSH | 81.62 | 82.91 |
Accuracy–latency comparison across settings. The accuracy–latency plot will show ALSH and Tandem under different settings alongside RelayGen and the other baselines. We will include this plot in the revised manuscript.
Motivation and alternatives
Thank you for this question. Below we explain how ALSH differs from the approaches you mentioned.
Weakness.
Question 1.
Response
A small model that cannot solve a question from scratch may still finish it from a large model's reasoning prefix. As discussed in Section 2, prior works like R2R, Hawkeye, Tandem and RelayGen have demonstrated efficiency gains from large–small collaboration within a request. Our work focuses on when to hand off reasoning and how to help the small model finish efficiently. We combine asynchronous handoff attempts with a finisher trained to complete partial reasoning prefixes.
Relative to request-level routing. At native 256k context, the same trained 2B model solves 40/200 OlymMATH questions alone (20.0%), whereas ALSH solves 149/200 (74.5%), with all 200 outputs committed by the finisher. The inherited reasoning creates completion opportunities unavailable to routing solely from the original question.
Relative to adaptive early stopping. Adaptive early stopping decides when the current model should stop reasoning and return an answer. ALSH instead lets a smaller model continue the remaining reasoning from the large model's intermediate state at a lower computational cost per token. Following your suggestion, we will include DEER as an adaptive early stopping baseline and report the results in the revised manuscript.
Relative to speculative decoding. Speculative decoding is orthogonal to ALSH. It speeds up token generation for a target model, while ALSH decides when to hand the remaining reasoning to a smaller model. It could therefore accelerate either the large model or the finisher in ALSH. We are also evaluating how much additional latency reduction can be achieved by combining ALSH with speculative decoding.
For example, adding DFlash to ALSH's large model further reduces mean latency from 60.73 to 41.59 s on AIME 2025. We will report the complete ten-benchmark comparisons in the revised manuscript.