- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GRACE, a framework for asynchronous SLM–LLM collaboration through graded representation augmentation. A context aligner synchronizes information between the models, while representation augmentation blocks incorporate LLM guidance into SLM decoding. Experiments evaluate instruction following, mathematical reasoning, inference efficiency, and cross-model collaboration.
Critical strengths
-
The paper proposes a practical framework for asynchronous SLM–LLM collaboration, incorporating LLM representations while preserving SLM autoregressive decoding. The graded augmentation and context synchronization mechanisms are clearly motivated and well designed.
-
The experiments demonstrate favorable performance–efficiency trade-offs on instruction-following and mathematical reasoning tasks. Component ablations, compute breakdowns, and cross-family evaluations provide additional evidence for the effectiveness of the approach.
Critical weaknesses
-
The interpretation of CNG* could be strengthened. CNG* divides accuracy gains by the absolute difference in inference computation. It is highly sensitive when this difference is close to zero. Table 2 reports a CNG* of 1276.00 on IFEval. The paper already reports accuracy, computation, and latency, and links this high value to reduced generation cost. A sensitivity analysis would clarify the interpretation of CNG* when the compute difference is small. Using the existing accuracy, total TFLOPs, and latency results to assess the same efficiency conclusions would strengthen this analysis.
-
The practical overhead of asynchronous collaboration deserves further analysis. Figure 4 evaluates robustness to simulated assistance delays, and Table 3 demonstrates the importance of KV cache reconstruction. Since reconstruction may introduce additional latency, further analysis of the time spent on cache reconstruction and context synchronization, and how much of this overhead is hidden by asynchronous execution, would provide a more complete assessment of deployment efficiency.
Questions for authors
- How sensitive are the reported CNG* values to small differences in computational cost? Do comparisons of accuracy against total TFLOPs and latency support the same relative efficiency advantages?
- During asynchronous inference, how much latency is introduced by KV cache reconstruction and context synchronization, and how much can be hidden through concurrent execution?
2 comments
Official Comment to Reviewer 54uj (Part 2)
W2, Q2: Latency Overhead of Asynchronous Inference
To measure the cost of each component of asynchronous collaboration, we ran a latency study on the longer prompts in MulDimIF (>512 tokens). We use Qwen3-1.7B as the SLM, with Qwen3-14B or Qwen3.5-9B as the LLM. The injection interval is 10 tokens. All experiments were run on NVIDIA RTX 5090 GPU. We report the latency of the first 100 decoded tokens to avoid interference from long-tail samples, averaged over 5 runs. We compare our asynchronous collaboration with a synchronous baseline. In the synchronous baseline, the SLM waits at each interaction for the LLM to finish its prefill/decode before it continues. In our asynchronous collaboration, the SLM keeps decoding while the LLM runs.
| Metric (ms) | Qwen3-1.7B + Qwen3-14B | Qwen3-1.7B + Qwen3.5-9B |
|---|---|---|
| Standalone decoding | ||
| SLM only | 1770.53 ± 0.36 | 1770.53 ± 0.36 |
| LLM only | 2612.52 ± 0.43 | 3361.78 ± 0.93 |
| Synchronous collaboration | ||
| LLM prefill (full prompt) | 110.49 ± 0.02 | 66.40 ± 0.00 |
| SLM cumulative waiting time | 415.80 ± 2.46 | 416.38 ± 5.75 |
| Total latency | 2343.80 ± 2.77 | 2441.76 ± 5.73 |
| Asynchronous collaboration (Ours) | ||
| KV cache reconstruction | 36.53 ± 0.01 | 36.97 ± 0.01 |
| Context sync (initial full synchronization) | 3.48 ± 0.01 | 3.60 ± 0.03 |
| Context sync (incremental synchronization) | 1.18 ± 0.00 | 1.17 ± 0.00 |
| Total latency | 1988.69 ± 1.06 | 2088.70 ± 0.34 |
| Δ Total latency (Ours − Sync) | −355.11 (−15.2%) | −353.06 (−14.5%) |
As shown in the table above, KV cache reconstruction and context synchronization both add only a small overhead. The main cost of asynchronous collaboration is the KV cache reconstruction that the SLM performs once at the first interaction, which takes about 37 ms. This is much cheaper than the full-prompt LLM prefill that synchronous collaboration must wait on, which takes 110.5 ms for Qwen3-14B and 66.4 ms for Qwen3.5-9B, or roughly 3.0× and 1.8× the reconstruction cost. After that, context synchronization is lightweight, taking about 3.5 ms for the initial full synchronization and about 1.2 ms for each incremental update. With all these overheads included, our method reduces end-to-end latency from 2343.8 ms to 1988.7 ms with Qwen3-14B and from 2441.8 ms to 2088.7 ms with Qwen3.5-9B, saving about 355 ms in both settings. Since the SLM accumulates about 416 ms of waiting time under synchronous collaboration, roughly 85% of this waiting cost is hidden by our asynchronous execution. Our method is also 23.9% and 37.9% faster than running Qwen3-14B and Qwen3.5-9B alone. Compared with SLM-only decoding, our method adds a moderate overhead. This is mainly because, in the current implementation, both models are deployed on a single GPU and compete for compute and memory bandwidth during concurrent execution. We plan to reduce this contention in future work through better scheduling and multi-GPU deployment.These results show that the core advantage of our design is that it replaces the costly wait for LLM prefill with a low-cost, one-time KV cache reconstruction on the SLM. The reconstruction cost depends only on the SLM, while the prefill cost grows with the size of the LLM, so we consider this advantage to become more pronounced with larger LLMs.
Official Comment to Reviewer 54uj (Part 1)
We sincerely appreciate your thorough and constructive feedback, which has been instrumental in improving the clarity and depth of our work. Below, we address your insightful comments in detail:
W1, Q1: Interpretation of CNG*
We thank the reviewer for this insightful question. We originally adopted CNG* to summarize the performance–efficiency trade-off of collaborative methods in a single, intuitive number. However, we agree that CNG* is sensitive to small differences in computational cost. To address this issue, we have replaced CNG* in the main tables with the raw inference TFLOPs, so that accuracy, compute, and latency can be compared directly. We have also added performance-compute Pareto plots for all SLM–LLM pairs and benchmarks (Figure 4), which show the trade-off more intuitively. Within each pair, GRACE lies on the Pareto frontier in most settings, achieving a favorable balance between accuracy and inference compute.
To further quantify the trade-off without the sensitivity of CNG*, we follow the strong/weak evaluation protocol of RouteLLM [1] and report the Performance Gap Recovered (PGR) [2] together with the relative inference cost:
PGR=AccLLM−AccSLMAccM−AccSLM×100
Cost=TFLOPsLLMTFLOPsM×100
PGR measures the fraction of the SLM–LLM accuracy gap recovered by the method, and Cost measures the fraction of the LLM’s inference compute it consumes. Neither quantity divides by a cost difference, so both remain well-behaved when a method’s cost is close to that of the SLM. The results below show that GRACE achieves a favorable performance–compute trade-off in most settings.
[1] Isaac Ong, et al., RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025
[2] Collin Burns, et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. ICML 2024