acceptodds
Under review as a conference paper at ICLR 2027

GRACE: Graded Representation Augmentation for Cost-Effective SLM–LLM Collaboration

Abstract

Large language models (LLMs) provide strong generation capabilities but incur substantial inference cost, while small language models (SLMs) offer efficient decoding at the expense of reduced performance on complex tasks. Recent studies have begun to explore ways of combining their complementary strengths, including speculative decoding and model switching. These efforts show the potential of involving stronger LLMs to assist SLM generation, but such assistance may still incur considerable additional computation and decoding latency. We propose GRACE, a Graded Representation Augmentation framework for Cost-Effective SLM-LLM collaboration, which improves SLM generation quality with minor additional LLM computation. GRACE establishes asynchronous bidirectional latent interaction between the frozen SLM and LLM, enabling LLM assistance without interrupting the SLM's autoregressive decoding. During decoding, a context aligner incrementally compresses the evolving SLM state into compact latent tokens to synchronize with the LLM, while representation augmentation blocks extract latent task-relevant guidance from the asynchronously available LLM representations to augment the SLM. These augmentations are performed at depth-aligned computation grades, allowing subsequent SLM layers to progressively absorb the guidance. Extensive experiments on instruction-following and mathematical benchmarks show that GRACE consistently improves SLM generation quality with favorable performance-efficiency trade-offs across different model scales, architectures, and versions, and remains effective when the collaborating models come from different model families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SLM-LLM Collaboration, Efficient Language Model Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_54ujOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GRACE, a framework for asynchronous SLM–LLM collaboration through graded representation augmentation. A context aligner synchronizes information between the models, while representation augmentation blocks incorporate LLM guidance into SLM decoding. Experiments evaluate instruction following, mathematical reasoning, inference efficiency, and cross-model collaboration.

Critical strengths
  1. The paper proposes a practical framework for asynchronous SLM–LLM collaboration, incorporating LLM representations while preserving SLM autoregressive decoding. The graded augmentation and context synchronization mechanisms are clearly motivated and well designed.

  2. The experiments demonstrate favorable performance–efficiency trade-offs on instruction-following and mathematical reasoning tasks. Component ablations, compute breakdowns, and cross-family evaluations provide additional evidence for the effectiveness of the approach.

Critical weaknesses
  1. The interpretation of CNG* could be strengthened. CNG* divides accuracy gains by the absolute difference in inference computation. It is highly sensitive when this difference is close to zero. Table 2 reports a CNG* of 1276.00 on IFEval. The paper already reports accuracy, computation, and latency, and links this high value to reduced generation cost. A sensitivity analysis would clarify the interpretation of CNG* when the compute difference is small. Using the existing accuracy, total TFLOPs, and latency results to assess the same efficiency conclusions would strengthen this analysis.

  2. The practical overhead of asynchronous collaboration deserves further analysis. Figure 4 evaluates robustness to simulated assistance delays, and Table 3 demonstrates the importance of KV cache reconstruction. Since reconstruction may introduce additional latency, further analysis of the time spent on cache reconstruction and context synchronization, and how much of this overhead is hidden by asynchronous execution, would provide a more complete assessment of deployment efficiency.

Questions for authors
  1. How sensitive are the reported CNG* values to small differences in computational cost? Do comparisons of accuracy against total TFLOPs and latency support the same relative efficiency advantages?
  2. During asynchronous inference, how much latency is introduced by KV cache reconstruction and context synchronization, and how much can be hidden through concurrent execution?
2 comments
AuthorsOpenReview ↗

Official Comment to Reviewer 54uj (Part 1)

We sincerely appreciate your thorough and constructive feedback, which has been instrumental in improving the clarity and depth of our work. Below, we address your insightful comments in detail:

W1, Q1: Interpretation of CNG*

We thank the reviewer for this insightful question. We originally adopted CNG* to summarize the performance–efficiency trade-off of collaborative methods in a single, intuitive number. However, we agree that CNG* is sensitive to small differences in computational cost. To address this issue, we have replaced CNG* in the main tables with the raw inference TFLOPs, so that accuracy, compute, and latency can be compared directly. We have also added performance-compute Pareto plots for all SLM–LLM pairs and benchmarks (Figure 4), which show the trade-off more intuitively. Within each pair, GRACE lies on the Pareto frontier in most settings, achieving a favorable balance between accuracy and inference compute.

To further quantify the trade-off without the sensitivity of CNG*, we follow the strong/weak evaluation protocol of RouteLLM [1] and report the Performance Gap Recovered (PGR) [2] together with the relative inference cost:

PGR=AccM−AccSLMAccLLM−AccSLM×100PGR=\frac{Acc_M-Acc_{SLM}}{Acc_{LLM}-Acc_{SLM}}\times 100 \\%,

Cost=TFLOPsMTFLOPsLLM×100Cost=\frac{TFLOPs_M}{TFLOPs_{LLM}}\times 100 \\%.

PGR measures the fraction of the SLM–LLM accuracy gap recovered by the method, and Cost measures the fraction of the LLM’s inference compute it consumes. Neither quantity divides by a cost difference, so both remain well-behaved when a method’s cost is close to that of the SLM. The results below show that GRACE achieves a favorable performance–compute trade-off in most settings.

SLM–LLM pair (PGR ↑ @ Cost↓)MethodIFEvalIFBenchLiveBench-IFMulDimIFAIME
Qwen3-1.7B + Qwen3-14BRouteLLM9.5 @ 22.225.9 @ 35.712.8 @ 22.167.8 @ 62.753.9 @ 69.1
Jev19.1 @ 31.325.9 @ 72.069.2 @ 55.818.8 @ 41.846.1 @ 82.0
C2C0.0 @ 31.122.2 @ 33.50.0 @ 71.344.7 @ 78.27.7 @ 21.9
GRACE (Ours)39.3 @ 18.477.8 @ 18.853.8 @ 29.880.6 @ 31.569.3 @ 16.7
Qwen3-1.7B + Qwen3-30B-A3BRouteLLM13.2 @ 48.116.7 @ 41.25.9 @ 56.566.6 @ 75.260.0 @ 79.2
Jev13.2 @ 49.113.9 @ 52.629.4 @ 64.113.8 @ 60.130.0 @ 92.0
C2C-10.8 @ 84.125.0 @ 75.3-20.6 @ 106.716.7 @ 141.0-10.0 @ 80.5
GRACE (Ours)25.3 @ 54.8102.7 @ 41.252.9 @ 87.175.7 @ 97.060.0 @ 45.4
Qwen3-1.7B + Qwen3.5-9BRouteLLM19.7 @ 31.919.6 @ 43.43.3 @ 31.360.7 @ 66.067.4 @ 69.0
Jev27.6 @ 39.436.9 @ 77.483.3 @ 61.717.0 @ 49.960.9 @ 89.9
GRACE (Ours)36.9 @ 29.863.0 @ 29.240.0 @ 46.474.2 @ 52.319.6 @ 15.6

[1] Isaac Ong, et al., RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025

[2] Collin Burns, et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. ICML 2024

AuthorsOpenReview ↗

Official Comment to Reviewer 54uj (Part 2)

W2, Q2: Latency Overhead of Asynchronous Inference

To measure the cost of each component of asynchronous collaboration, we ran a latency study on the longer prompts in MulDimIF (>512 tokens). We use Qwen3-1.7B as the SLM, with Qwen3-14B or Qwen3.5-9B as the LLM. The injection interval is 10 tokens. All experiments were run on NVIDIA RTX 5090 GPU. We report the latency of the first 100 decoded tokens to avoid interference from long-tail samples, averaged over 5 runs. We compare our asynchronous collaboration with a synchronous baseline. In the synchronous baseline, the SLM waits at each interaction for the LLM to finish its prefill/decode before it continues. In our asynchronous collaboration, the SLM keeps decoding while the LLM runs.

Metric (ms)Qwen3-1.7B + Qwen3-14BQwen3-1.7B + Qwen3.5-9B
Standalone decoding
SLM only1770.53 ± 0.361770.53 ± 0.36
LLM only2612.52 ± 0.433361.78 ± 0.93
Synchronous collaboration
LLM prefill (full prompt)110.49 ± 0.0266.40 ± 0.00
SLM cumulative waiting time415.80 ± 2.46416.38 ± 5.75
Total latency2343.80 ± 2.772441.76 ± 5.73
Asynchronous collaboration (Ours)
KV cache reconstruction36.53 ± 0.0136.97 ± 0.01
Context sync (initial full synchronization)3.48 ± 0.013.60 ± 0.03
Context sync (incremental synchronization)1.18 ± 0.001.17 ± 0.00
Total latency1988.69 ± 1.062088.70 ± 0.34
Δ Total latency (Ours − Sync)−355.11 (−15.2%)−353.06 (−14.5%)

As shown in the table above, KV cache reconstruction and context synchronization both add only a small overhead. The main cost of asynchronous collaboration is the KV cache reconstruction that the SLM performs once at the first interaction, which takes about 37 ms. This is much cheaper than the full-prompt LLM prefill that synchronous collaboration must wait on, which takes 110.5 ms for Qwen3-14B and 66.4 ms for Qwen3.5-9B, or roughly 3.0× and 1.8× the reconstruction cost. After that, context synchronization is lightweight, taking about 3.5 ms for the initial full synchronization and about 1.2 ms for each incremental update. With all these overheads included, our method reduces end-to-end latency from 2343.8 ms to 1988.7 ms with Qwen3-14B and from 2441.8 ms to 2088.7 ms with Qwen3.5-9B, saving about 355 ms in both settings. Since the SLM accumulates about 416 ms of waiting time under synchronous collaboration, roughly 85% of this waiting cost is hidden by our asynchronous execution. Our method is also 23.9% and 37.9% faster than running Qwen3-14B and Qwen3.5-9B alone. Compared with SLM-only decoding, our method adds a moderate overhead. This is mainly because, in the current implementation, both models are deployed on a single GPU and compete for compute and memory bandwidth during concurrent execution. We plan to reduce this contention in future work through better scheduling and multi-GPU deployment.These results show that the core advantage of our design is that it replaces the costly wait for LLM prefill with a low-cost, one-time KV cache reconstruction on the SLM. The reconstruction cost depends only on the SLM, while the prefill cost grows with the size of the LLM, so we consider this advantage to become more pronounced with larger LLMs.

Reviewer_g9HMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRACE, a framework in which a frozen SLM remains the sole autoregressive decoder while a frozen LLM asynchronously provides latent guidance at depth-aligned "computation grades". A context aligner compresses SLM states into LLM-readable latent tokens, and representation augmentation blocks inject cross-attended LLM representations back into the SLM. Experiments on instruction-following benchmarks and AIME 2024–2026 with several Qwen, Ministral and Gemma pairings report improved SLM accuracy at a small additional compute cost compared with EAGLE-3, RouteLLM, Jev and C2C.

Critical strengths

Speculative decoding verifies drafts with the target LLM, and routing hands generation over to the LLM. GRACE instead treats LLM hidden states as asynchronous, depth-aligned "guidance" injected into a frozen SLM. Both backbones stay frozen, and the latent sync also works across tokenizers. This makes the idea practically attractive and conceptually distinct from prior SLM–LLM collaboration.

Critical weaknesses

GRACE trains a context aligner and RABs at three grades, which together inject learned residuals into the SLM. The training data are closely related to the evaluation benchmarks. Part of the improvement might therefore come from in-domain adaptation of the SLM through the added modules rather than from LLM representations. The large gains on the pretrained-only Gemma4-E2B make this question especially relevant. The central claim of the paper is that "LLM representations carry task-relevant guidance", so the following control is necessary to support it: a parameter-matched LoRA/SFT of the SLM on the same data. Relatedly, could the authors clarify how the "pair-specific supervision" / gold responses y are constructed? If they are generated by the LLM, the training also contains a distillation component, and an SFT-on-LLM-outputs baseline would then be the natural comparison.

In GRACE, the LLM never sees the original prompt text, it only receives latent tokens compressed from SLM final-layer states. Table 4 suggests this channel loses a fair amount of information: when the LLM is conditioned only on latent context, IFEval drops from 83.55 to 69.50 and LiveBench-IF from 69.50 to 56.00. This seems to bound how much the LLM can contribute. The Qwen3.5-9B AIME result may illustrate this: the LLM reaches 58.89, but GRACE reaches 17.78. I am curious why the design does not let the LLM prefill the raw prompt directly, which is parallelizable and already asynchronous to SLM decoding, and use latent sync only for the generated tokens. A discussion of this design choice, together with a comparison against a "raw-prompt prefill + latent sync for generated tokens" variant, would help clarify where the bottleneck lies.

Questions for authors

For the first interaction, the query is ΦQ(SelfAttn(HS))\Phi_Q(\mathrm{SelfAttn}(H^S)) (Sec. 3.2). What is the motivation for the additional self-attention, as opposed to using HSH^S directly? In addition, RABs attend to all LLM tokens of grade kk. How do memory and compute scale with long generations, and would a recent window suffice? Relatedly, CALM (Bansal et al., 2024) also composes two frozen models through learned cross-attention between their intermediate layer representations. It would be helpful to discuss the distinction, e.g., GRACE's asynchronous execution, latent context synchronization and graded depth alignment. If feasible, a CALM-style synchronous variant would also show how much the asynchronous design trades off in quality.

The proportional depth partition is motivated by t-SNE overlap (Fig. 5), which is fairly qualitative evidence. Did the authors consider a similarity-based alignment (Kornblith et al., 2019)? Is the proportional choice close to optimal under such a measure? Also, which layer within each LLM grade provides Ht,kLH^L_{t,k}? Why is augmentation applied only at the first K−1K-1 grades and not at GKG_K?

The "no-wait" property (ΔTwait=0\Delta T_{wait} = 0) is a core claim but is not directly measured. At the first interaction, GRACE replays the accumulated SLM context and rebuilds the augmented KV cache, and the paper does not state whether SLM decoding pauses during this replay or how often subsequent interactions are triggered. The paper should report a wall-clock breakdown (SLM decoding, replay/KV reconstruction, LLM-side computation) and the actual delay dd observed when both models share one GPU. In addition, EAGLE-3 is run with vLLM while the other methods use a different stack, so the latency numbers in Table 1 are not directly comparable.

CNG* divides by the absolute compute difference, so it explodes when GRACE's cost is close to the SLM's (e.g., 1276.00 on IFEval in Table 2 and 64.82 on AIME in Table 1), and it does not distinguish cheaper from more expensive methods. These values overstate the efficiency advantage. An accuracy-vs-TFLOPs Pareto plot would present the trade-off more reliably.

Two reported numbers should be double-checked. Under greedy decoding on 30 problems per year, AIME scores should be multiples of 1/30, yet RouteLLM's scores in Table 10 are 16.77 and 23.30, and these values are used in the 15.58 average in Table 1.

References:

  • Bansal R, Samanta B, Dalmia S, et al. Llm augmented llms: Expanding capabilities through composition[C]//International Conference on Learning Representations. 2024.
  • Kornblith S, Norouzi M, Lee H, et al. Similarity of neural network representations revisited[C]//International conference on machine learning. PMlR, 2019: 3519-3529.
8 comments
AuthorsOpenReview ↗

Official Comment to Reviewer g9HM (Part 1)

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

W1: Disentangling LLM guidance from in-domain adaptation and distillation.

We address this concern from two aspects: (1) whether GRACE genuinely leverages task-relevant guidance from the LLM, and (2) how GRACE compares with a parameter-matched SFT baseline trained on the same LLM-generated data.

(1) GRACE relies on task-relevant guidance from the LLM. If the gains came mainly from in-domain adaptation through the added modules, the learned residuals should still help when the LLM input carries no information about the current task. To test this, we ran perturbation experiments on the Qwen3-1.7B / Qwen3-14B pair: we replaced only the LLM-side input and left the SLM input and all trained modules unchanged. We consider three settings. In Same-task Mismatched, the LLM receives a different query randomly sampled from the same benchmark. In Fixed Off-task, it receives a single unrelated prompt shared across all evaluation queries (e.g., a math problem for an instruction-following query). In Random Off-task, it receives a randomly sampled unrelated prompt for each query. In all three settings, the substituted content is compressed by the context aligner into latent context tokens in the usual way, and all other evaluation settings, including the 2-token simulated delay, are the same as for GRACE.

ConfigurationIFEval (P-Acc. ↑)IFBench(P-Acc. ↑)LiveBench-IF (P-Acc. ↑)MulDimIF (P-Acc.↑)AIME (Acc. ↑)
Qwen3-1.7B68.0217.6750.0032.927.78
Qwen3-14B83.5526.6769.5059.5822.22
GRACE(Ours)74.1224.6760.5054.4217.78
Same-task Mismatched62.1115.3352.5036.257.78
Fixed Off-task Input60.2616.6744.0029.587.78
Random Off-task Input61.1818.0045.0028.925.56

All three perturbations remove GRACE's gains: on every benchmark, accuracy falls 6.67 to 25.50 points below GRACE with matched input. With off-task input, accuracy is at or below the standalone SLM in 9 of 10 cases, for example IFEval drops from 68.02 to 60.26. With same-task mismatched input, only a small residual gain survives on LiveBench-IF and MulDimIF, about 24% and 15% of GRACE's gains there, which we attribute to task-level information shared within a benchmark. An input-independent in-domain shift would be insensitive to these substitutions. These results indicate that GRACE's gains are driven mainly by query-specific guidance from the LLM rather than by an input-independent adaptation of the SLM.

(2) Construction of yy, and comparison with parameter-matched SFT on LLM outputs. Although the gold responses yy are generated by the corresponding paired LLM, GRACE and LoRA improve the SLM in two different ways. GRACE keeps both the SLM and the LLM frozen and improves the SLM through collaboration at inference time, whereas LoRA fine-tunes the SLM by training low-rank adapters on task-specific supervision. Following the reviewer’s suggestion, we fine-tuned Qwen3-1.7B with LoRA on exactly the same LLM-generated data. LoRA adapters are applied to all attention projections (q, k, v, o) and FFN projections (gate, up, down) in every Transformer layer, with the rank set to 624, so that the number of trainable parameters matches that of GRACE’s RABs. We also evaluated a combined setting: we froze the LoRA-tuned SLM and trained GRACE’s collaboration modules on top of it.

ConfigurationIFEval (P-Acc.↑)IFBench (P-Acc.↑)LiveBench-IF (P-Acc.↑)MulDimIF (P-Acc.↑)AIME (Acc.↑)
Qwen3-1.7B68.0217.6750.0032.927.78
Qwen3-14B83.5526.6769.5059.5822.22
GRACE74.1224.6760.5054.4217.78
LoRA74.1217.6759.0055.9212.22
LoRA + GRACE76.3423.6763.5055.9216.67

(Continued on next comment)

AuthorsOpenReview ↗

Official Comment to Reviewer g9HM (Part 2)

W1: Disentangling LLM guidance from in-domain adaptation and distillation.(Continued)

As shown in the table above, LoRA matches GRACE on IFEval and MulDimIF, whose constraint types are covered by the training data, and is close on LiveBench-IF. On the more challenging benchmarks, however, its improvement is limited: it brings no gain on IFBench with its out-of-distribution constraints, and recovers less than half of GRACE's gain on AIME. We attribute this to what each method can use. Distilling into the weights captures the training distribution, while GRACE reads the LLM’s representation of each specific query at inference time, which appears to generalize better to unseen constraints and harder problems. Moreover, we can also use our collaboration paradigm to enhance the performance of LoRA-tuned SLM. Adding GRACE on top of the frozen LoRA-tuned SLM brings further gains: +2.22 on IFEval, +6.00 on IFBench, +4.50 on LiveBench-IF, and +4.45 on AIME. Since the LoRA-tuned SLM has already been adapted to the same data, these additional gains come from the LLM guidance rather than from in-domain adaptation. MulDimIF stays the same because LoRA-tuned SLM already closes much of the gap to the LLM, leaving limited headroom.

In summary, our perturbation experiments demonstrate that GRACE’s gains depend on task-relevant LLM representations. Furthermore, GRACE and LoRA represent two distinct technical routes for improving SLMs. GRACE enhances the SLM through collaboration while keeping both models frozen, whereas LoRA directly updates the SLM’s parameters. GRACE can be readily applied on top of a LoRA-tuned SLM to further boost its performance.


W2: Motivation for synchronizing the prompt through latent tokens instead of the raw prompt

We thank the reviewer for this insightful question. Our motivation for using latent context tokens throughout GRACE is threefold.

(1) We intend the context aligner to serve as a learnable interface rather than mere context synchronization. Beyond passing the context to the LLM, the context aligner maps the SLM’s decoding states into soft tokens that the frozen LLM can read. Because it is trained jointly with the RABs, the aligner is optimized not only to keep the LLM informed of the context, but also to encourage the LLM to produce representations that the SLM can more readily integrate as guidance. We therefore regard the latent context as part of the collaboration mechanism itself, rather than as a substitute for the raw prompt. From this perspective, Table 4 serves as a probing study. Its purpose is to verify that the latent tokens remain interpretable to the frozen LLM and retain task-relevant context. It is not intended as an upper bound on what GRACE can achieve. In GRACE, the SLM still reads the raw prompt directly, and the LLM provides complementary guidance rather than solving the task on its own. Consistent with this view, GRACE performs comparably to or better than the latent-only LLM across the instruction-following benchmarks.

(2) Latent compression substantially reduces LLM-side computation. In GRACE, the LLM's cost largely comes from prefilling the context. The context aligner compresses each buffer of b SLM states into a single latent token, which shortens the sequence the LLM processes. This saving grows with prompt length. In the probing study of Table 4, the total LLM inference cost drops from 21.34 to 14.00 TFLOPs on LiveBench-IF and from 15.70 to 10.03 TFLOPs on MulDimIF when the text context is replaced by latent tokens. Keeping the LLM-side cost low is central to the favorable performance–efficiency trade-off that GRACE targets.

(3) Raw text never needs to leave the SLM side, which potentially helps preserve data privacy. For our experiments, we run the SLM and LLM on the same GPU. As the LLM scales further, however, the two models will need to be deployed on separate GPUs or even separate devices. In such settings, synchronizing only latent context tokens means the user’s raw prompt and generated text never have to be transmitted in plaintext. We see this as a potential benefit of the latent interface for collaborative deployment.

(Continued on next comment)

AuthorsOpenReview ↗

Official Comment to Reviewer g9HM (Part 3)

W2: Motivation for synchronizing the prompt through latent tokens instead of the raw prompt (Continued)

Following the reviewer’s suggestion, we also explored the variant, "raw-prompt prefill + latent sync for generated tokens". GRACE is trained with teacher forcing on gold prefixes, where the RABs learn to interact with the complete latent context synchronized from the prompt, and subsequent interactions at inference rely on generalization. The main challenge of this variant is that the RABs must adapt to two types of input at once: raw text tokens for the prompt and latent context tokens for the generated content. We tried two training strategies. The first mixes the two forms of context within each sample. We keep a certain portion of the context as raw text and fill in the rest with latent context tokens. With this strategy, we observed attention-sink-like behavior, where the model concentrates its attention on the text tokens and largely ignores the latent portion, which leads to poor performance. The second randomly switches between raw text and latent context tokens during training, so that the RABs learn to handle both forms of input. The table below reports this second strategy.

Variant (P-Acc.↑/TFLOPs↓)IFEvalIFBenchLiveBench-IFMulDimIF
Qwen3-1.7B68.02 / 1.2617.67 / 1.6550.00 / 2.5432.92 / 1.78
Qwen3-14B83.55 / 10.5526.67 / 12.9169.50 / 21.3459.58 / 15.70
Raw-prompt prefill + latent sync74.49 / 4.4220.33 / 5.8554.50 / 14.5146.75 / 10.16
GRACE (all latent)74.12 / 1.9424.67 / 2.4360.50 / 6.3654.42 / 4.95

Under our current training setup, this variant already shows promising potential: it consistently improves over the standalone SLM on all four benchmarks and performs comparably to GRACE on IFEval. At the same time, its computation increases notably, since the LLM needs to prefill the full raw prompt. On the other benchmarks, a gap to GRACE remains, which we attribute to two factors. First, our current training strategy is relatively simple and may not yet allow the RABs to adapt well to switching between raw text and latent context tokens. Second, when the prompt is passed as raw text, that part of the context no longer goes through the context aligner, so the variant loses the benefit of the learnable interface described in (1). We plan to explore better training strategies for this hybrid setting in future work.


Q1: Motivation for the additional self-attention in the first interaction

We introduced the self-attention based on two considerations. First, RAB is responsible for both aligning the SLM and LLM representations and extracting the latent guidance from the LLM. We were concerned that cross-attention alone might have limited capacity to handle both functions, so we added a self-attention to refine the SLM representations before they are used as queries. Second, the frozen SLM representations are produced by causal attention, so an early position only encodes its prefix and is unaware of later content. The self-attention lets each position aggregate information over the whole replayed sequence before querying the LLM representations, so that the retrieved guidance is consistent with the global task understanding. As suggested, we ablate the self-attention by directly using ΦQ(Ht+d,kS)\Phi_Q(H^S_{t+d,k}) as queries on Qwen3-1.7B and Qwen3-14B pair. Removing the self-attention leads to a performance drop across benchmarks, suggesting that it is beneficial for the initial interaction.

VariantIFEval (P-Acc.↑)IFBench (P-Acc.↑)LiveBench-IF (P-Acc.↑)MulDimIF (P-Acc.↑)
GRACE w/o SelfAttn71.9020.3353.5044.75
GRACE (Full)74.1224.6760.5054.42
Reviewer_3CqsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a methodology (GRACE) for augmenting small language models (SLMs) with LLM computation to improve task-performance. GRACE composes of a context aligner and representation augmentation blocks (RABs). The context aligner compresses the hidden states of the SLM into tokens that are in the vocabulary of the LLM, representing the current context of the SLM during decoding. The RABs takes sets of SLM and LLM blocks and produces a representation for the next SLM block; the goal is to incorporate LLM compute, guided by the current SLM generation as context, into the following SLM computation.

Critical strengths

The evaluation demonstrates GRACE produces improved accuracy over the base SLM and other SLM+LLM augmentation methods, while having less latency than the standalone latency of LLM model used to augment SLM. It also uses less TFLOPs per query than the LLM and other SLM+LLM augmentation methods. GRACE is a new point in the accuracy to latency and compute tradeoff.

Critical weaknesses

The paper is difficult to follow at points, requiring multiple rereads to understand how the method works. In particular, the author introduces several terminology/jargon that is not defined until later (e.g. "asynchronous bidirectional latent interaction", "depth-aligned computation grades", "absorb the guidance"). The one-off reference to human brain processing seems tenuous, and it's not clear what the connection there is or how it contributes to the methodology.

Questions for authors
  • Is my understanding correct that SLM does not need to wait for LLM model, but continuously generates tokens with the LLM model running in the background and incorporating its output when it gets them? And since the SLM model itself is frozen, not receiving RAB outputs doesn't harm the SLM model, only that it doesn't get any benefits?
  • Do the authors have any explanations for why using the bigger Qwen3-30B model as the LLM in GRACE does not perform better than using the smaller Qwen3-14B model as the LLM?
  • What is the distribution of dd (the token delay) during inference?
  • How much training is required
  • Could the authors explain how to interpret CNG?
  • How does the choice of KK affect performance?
6 comments
AuthorsOpenReview ↗

Official Comment to Reviewer 3Cqs (Part 1)

We sincerely appreciate your valuable feedback, which has greatly contributed to improving the clarity and robustness of our work. Below, we address each of your insightful comments in detail:

W1: Clarification of Core Terminology

We have revised the paper to clarify these core terms. Below, we clarify each term in detail:

  • Asynchronous: The SLM executes autoregressive token decoding continuously without ever stalling to wait for the LLM. The LLM runs concurrently in the background. Whenever LLM intermediate representations become available, the SLM incorporates them into its current step.
  • Bidirectional: Information flows between the two models in a closed loop. For SLM to LLM direction, the Context Aligner incrementally compresses consecutive windows of SLM hidden states and appends them to the LLM's context, keeping the LLM updated on the generation history without text re-encoding. For LLM to SLM direction, Representation Augmentation Blocks (RABs) extract guidance from the LLM’s intermediate states and inject it directly into the SLM’s hidden states.
  • Latent: Both directions exchange continuous hidden representations rather than discrete text tokens.
  • Depth-aligned computation grades: We split each model's transformer blocks into K = 4 groups of consecutive layers at the same relative depth, and pair the k-th group of the SLM with the k-th group of the LLM. For example, the first 7 of the 28 Qwen3-1.7B blocks are paired with the first 10 of the 40 Qwen3-14B blocks. Each pair is a "grade". We have revised this phrase to computation grades paired at corresponding layer depths.
  • Absorb the guidance: At the boundary of SLM grade, the guidance produced by the RAB is injected directly into the SLM hidden state via a residual connection. The subsequent frozen SLM Transformer blocks then process this modified hidden state during standard forward propagation, progressively integrating the LLM's guidance before the next grade's guidance is injected. We have revised the word absorb to the word integrate.

W2: Clarification of the Neuroscientific Motivation Behind the Graded Design

When conceptualizing GRACE, our graded collaboration was inspired by recent neuroscientific findings[1], which showed that the human brain encodes spatial novelty through graded representations along hippocampal and cortical hierarchies rather than through abrupt state transitions. Instead of forcing the SLM to integrate all LLM guidance abruptly at a single layer, GRACE introduces latent guidance progressively across grades. This enables intermediate SLM layers to naturally integrate earlier LLM guidance before deeper grades inject complementary refinements. The design is also supported empirically: in the grade ablation of Table 5, using all three grades improves over the best single grade by 5.56 to 16.75 points. In the revised manuscript, we have rephrased this discussion to present this reference as an inspiration and to point to the ablation.

[1] Quent J A, et al. Graded encoding of spatial novelty scales in the human brain[J]. Nature Communications, 2025


Q1: Confirmation on Non-Blocking Asynchronous Decoding

Yes, the reviewer’s understanding is completely correct on both points. Our asynchronous design fully decouples the execution of the two models, ensuring the SLM decodes tokens continuously without ever pausing to wait for the LLM. Because both the SLM and LLM are strictly frozen and interact via residual connections, not using the RAB outputs does not harm any capabilities of the standalone SLM.

AuthorsOpenReview ↗

Official Comment to Reviewer 3Cqs (Part 2)

Q2: Performance comparison between Qwen3-30B-A3B and Qwen3-14B within GRACE.

To thoroughly investigate this phenomenon, we re-evaluated both the standalone baseline of Qwen3-30B-A3B and its collaborative performance with GRACE. As shown in the table below, although Qwen3-30B-A3B has a larger total parameter scale, it adopts a Mixture-of-Experts (MoE) architecture with only 3B activated parameters per token, so its standalone benchmark capability is actually comparable to or even slightly weaker than that of the dense Qwen3-14B model. Specifically, the standalone Qwen3-14B achieves 69.50 on LiveBench-IF, 59.58 on MulDimIF, and an average accuracy of 22.22 across AIME 2024 to 2026, surpassing the standalone Qwen3-30B-A3B which reaches 67.00, 58.42, and 21.11 on these benchmarks. Because the benefit of collaboration is largely determined by the actual capability of the target LLM, this similar standalone strength explains why their collaborative results under GRACE appear close across most tasks. Conversely, on IFBench, where the standalone Qwen3-30B-A3B is stronger than Qwen3-14B, GRACE (Qwen3-1.7B + Qwen3-30B-A3B) also outperforms GRACE (Qwen3-1.7B + Qwen3-14B), consistent with GRACE benefiting from a stronger target LLM.

MethodsIFEval (P-Acc.↑)IFBench (P-Acc.↑)LiveBench-IF (P-Acc.↑)MulDimIF (P-Acc.↑)AIME (Acc.↑)
Qwen3-1.7B68.0217.6750.0032.927.78
Qwen3-14B83.5526.6769.5059.5822.22
Qwen3-30B-A3B84.1028.0067.0058.4221.11
GRACE(Qwen3-1.7B + Qwen3-14B)74.1224.6760.5054.4217.78
GRACE(Qwen3-1.7B + Qwen3-30B-A3B)72.6428.3359.0052.2516.67

Q3: Distribution of Token Delay During Inference

To evaluate the token delay during inference, we conducted tests on MulDimIF, which features a wide sample length distribution, across different SLM–LLM pairs over 5 repeated runs, with the detailed latency metrics summarized in the table below where Signal Ready Latency denotes the elapsed time for the LLM to obtain the corresponding guidance representation, and First Injection Latency denotes the elapsed time before the SLM begins incorporating guidance.

Model PairMetric< 256 tokens256–512 tokens> 512 tokens
Qwen3-1.7B + Qwen3-14BSignal Ready Latency (ms)56.12 ±\pm 0.8371.41 ±\pm 6.8867.02 ±\pm 1.89
First Injection Latency (ms)74.18 ±\pm 0.8688.42 ±\pm 5.8583.27 ±\pm 2.07
Mean Delay (tokens)1.998 ±\pm 0.0012.000 ±\pm 0.0002.118 ±\pm 0.039
Qwen3-1.7B + Qwen3.5-9BSignal Ready Latency (ms)75.45 ±\pm 1.2397.45 ±\pm 9.8077.91 ±\pm 1.55
First Injection Latency (ms)92.68 ±\pm 0.99115.11 ±\pm 10.0094.03 ±\pm 1.73
Mean Delay (tokens)2.000 ±\pm 0.0011.998 ±\pm 0.0042.000 ±\pm 0.000

In wall-clock time, the LLM’s guidance signal is ready within 56–97 ms, and the SLM begins incorporating it within 74–115 ms across all length regimes and both model pairs. The gap between Signal Ready Latency and First Injection Latency stays at about 16–18 ms in every setting, which is roughly one SLM decoding step. So once the guidance is ready, the SLM uses it at the very next decoding step, with no extra synchronization overhead. In tokens, the mean delay stays at about 2 tokens in all settings.


Q4: Training Requirements

Throughout training, both the SLM and LLM backbones remain frozen, and only the context aligner and representation augmentation blocks are optimized. The training pipeline consists of two stages: we first warm up the Context Aligner for 15 epochs, and subsequently conduct collaborative training for 10 epochs under a simulated asynchronous delay. For training data, we construct 60,000 supervision trajectories generated by the frozen LLM over IF-RLVR, RLVR-IFEval, and the training split of MulDimIF. For mathematical reasoning, we further fine-tune for 10 epochs on 20,000 NuminaMath-1.5 examples. More implementation details are illustrated in Appendix B.

AuthorsOpenReview ↗

Official Comment to Reviewer 3Cqs (Part 3)

Q5: Interpretation of CNG*

We thank the reviewer for this meaningful question. We originally adopted CNG* to summarize the performance–efficiency trade-off of collaborative methods in a single, intuitive number. It measures the accuracy gain over the SLM per TFLOP of compute difference, so a higher value means a more compute-efficient gain. However, we recognize that CNG* is sensitive to small differences in computational cost. To address this issue, we have replaced CNG* in the main tables with the raw inference TFLOPs, so that accuracy, compute, and latency can be compared directly. We have also added performance-compute Pareto plots for all SLM–LLM pairs and benchmarks (Figure 4), which show the trade-off more intuitively. Within each pair, GRACE lies on the Pareto frontier in most settings, achieving a favorable balance between accuracy and inference compute.

To further quantify the trade-off without the sensitivity of CNG*, we follow the strong/weak evaluation protocol of RouteLLM [1] and report the Performance Gap Recovered (PGR) [2] together with the relative inference cost:

PGR=AccM−AccSLMAccLLM−AccSLM×100PGR=\frac{Acc_M-Acc_{SLM}}{Acc_{LLM}-Acc_{SLM}}\times 100 \\%,

Cost=TFLOPsMTFLOPsLLM×100Cost=\frac{TFLOPs_M}{TFLOPs_{LLM}}\times 100 \\%.

PGR measures the fraction of the SLM–LLM accuracy gap recovered by the method, and Cost measures the fraction of the LLM’s inference compute it consumes. Neither quantity divides by a cost difference, so both remain well-behaved when a method’s cost is close to that of the SLM. The results below show that GRACE achieves a favorable performance–compute trade-off in most settings.

SLM–LLM pair (PGR ↑ @ Cost↓)MethodIFEvalIFBenchLiveBench-IFMulDimIFAIME
Qwen3-1.7B + Qwen3-14BRouteLLM9.5 @ 22.225.9 @ 35.712.8 @ 22.167.8 @ 62.753.9 @ 69.1
Jev19.1 @ 31.325.9 @ 72.069.2 @ 55.818.8 @ 41.846.1 @ 82.0
C2C0.0 @ 31.122.2 @ 33.50.0 @ 71.344.7 @ 78.27.7 @ 21.9
GRACE (Ours)39.3 @ 18.477.8 @ 18.853.8 @ 29.880.6 @ 31.569.3 @ 16.7
Qwen3-1.7B + Qwen3-30B-A3BRouteLLM13.2 @ 48.116.7 @ 41.25.9 @ 56.566.6 @ 75.260.0 @ 79.2
Jev13.2 @ 49.113.9 @ 52.629.4 @ 64.113.8 @ 60.130.0 @ 92.0
C2C-10.8 @ 84.125.0 @ 75.3-20.6 @ 106.716.7 @ 141.0-10.0 @ 80.5
GRACE (Ours)25.3 @ 54.8102.7 @ 41.252.9 @ 87.175.7 @ 97.060.0 @ 45.4
Qwen3-1.7B + Qwen3.5-9BRouteLLM19.7 @ 31.919.6 @ 43.43.3 @ 31.360.7 @ 66.067.4 @ 69.0
Jev27.6 @ 39.436.9 @ 77.483.3 @ 61.717.0 @ 49.960.9 @ 89.9
GRACE (Ours)36.9 @ 29.863.0 @ 29.240.0 @ 46.474.2 @ 52.319.6 @ 15.6

[1] Isaac Ong, et al., RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025

[2] Collin Burns, et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. ICML 2024

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are grateful that the reviewers found GRACE "practically attractive and conceptually distinct from prior SLM–LLM collaboration" (Reviewer g9HM), with mechanisms that are "clearly motivated and well designed" (Reviewer 54uj), and recognized it as "a new point in the accuracy to latency and compute tradeoff" (Reviewer 3Cqs).

We have carefully addressed all the constructive feedback provided by the reviewers. The main additions are:

  1. GRACE's gains rely on LLM's task-relevant guidance. Our perturbation experiments replace only the LLM-side input with mismatched or off-task content, keeping the SLM input and all trained modules unchanged. These perturbations demonstrates that the gains come from query-specific LLM guidance rather than from in-domain adaptation of the SLM.
  2. The no-wait property is directly measured. On a single GPU, the measured delay stays at about 2 tokens, and GRACE reduces end-to-end latency by about 15% compared with synchronous collaboration.
  3. Performance-efficiency trade-off is evaluated with robust metrics. We replaced CNG* with raw TFLOPs, added accuracy–compute Pareto plots, and report PGR together with relative cost. GRACE lies on the Pareto frontier in most settings. Compared with a synchronous CALM-style variant, GRACE retains 50% to 87% of its accuracy gain with 16% to 28% of its compute.
  4. Design choices are further analyzed. We added CKA-based layer alignment, the effect of K, sliding-window augmentation, a self-attention ablation, a raw-prompt prefill variant, and clarified the core terminology.

In summary, GRACE enables a frozen LLM to assist a frozen SLM through asynchronous latent guidance, improving generation quality without interrupting SLM decoding and with minor additional LLM computation. We hope our responses address the reviewers' concerns and would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.