acceptodds
Under review as a conference paper at ICLR 2027

SLMCoach: Context-Adaptive LM-Head Caching with Cross-Turn Reuse for Efficient Small Language Model Inference

Abstract

Small language models (SLMs) have emerged as a promising solution for privacy-preserving, low-latency artificial intelligence on edge devices. However, SLM inference remains constrained by memory footprint and data-movement overhead, with the vocabulary-dependent language modeling head (LM-head) constituting a substantial bottleneck. Existing vocabulary-reduction methods reduce these costs but typically treat queries independently, overlooking the temporal locality inherent in multi-turn interactions and missing opportunities to reuse LM-head rows across turns. To address this limitation, we formulate the cross-turn LM-head caching problem and introduce SLMCoach, a context-adaptive framework for efficient multi-turn SLM inference. By exploiting contextual continuity across dialogue turns, SLMCoach selectively retains LM-head rows with high reuse potential and incrementally updates a budget-constrained cache, reducing redundant parameter movement across consecutive queries. Empirical evaluations on a multi-turn dialogue benchmark demonstrate that SLMCoach reduces the LM-head footprint by 99.2% and peak GPU memory by 51.5% relative to full-head inference while preserving response quality. At matched response fidelity, SLMCoach further reduces LM-head data movement by 7.5× compared with query-isolated loading. These results highlight the benefit of exploiting cross-turn reuse for resource-constrained SLM inference.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Small Language Models, Context-Adaptive Caching, LM-Head Caching, Efficient Language Model Inference, Multi-Turn Dialogue

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pm9TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SLMCoach, a context-adaptive LM-head cache for multi-turn small language model inference. It combines offline vocabulary profiling, dialogue-dependent relevance and prevalence, and residency-aware selection to retain useful output rows on the GPU and transfer only missing rows between turns. Experiments on MultiWOZ with Gemma-3-270M-IT report substantial memory and transfer reductions, with broadly similar outputs to full-head inference. Comparisons across cache budgets and serving modes, together with component ablations, indicate that explicit reuse is most beneficial under KV-cached serving and tight budgets.

Critical strengths
  • The paper targets a practical, underexplored source of inference overhead by exploiting cross-turn locality in vocabulary-dependent weights.
  • The framework coherently combines dialogue relevance, recent token usage, and cache residency to balance output coverage against parameter transfers.
  • The experiments demonstrate substantial LM-head memory and transfer savings while maintaining broadly comparable response fidelity to full-head inference.
Critical weaknesses
  • Evaluation is confined to a narrowly scoped dialogue benchmark, a compact model, and desktop GPU hardware, leaving broader model and edge-deployment applicability insufficiently established.

  • Quality evaluation measures agreement with full-head outputs under greedy decoding rather than task success or human judgments, limiting support for the claim of preserved response quality.

  • The cache policy requires all serving-context token rows to be mandatory but does not explain how the strict memory budget is maintained when this set exceeds capacity.

  • End-to-end latency is essentially unchanged against the strongest query-isolated baseline, while selection overhead and auxiliary metadata memory are not separately quantified.

Questions for authors
  • How is the cache budget enforced when mandatory serving-context rows exceed capacity, and does the evaluation cover such cases under either serving mode?
  • Task-oriented quality or human assessment alongside full-head agreement should be reported, and whether the fidelity conclusions extend beyond greedy decoding?
  • The memory–quality trade-off should be validated on another model or in a less constrained dialogue setting, and the behavior of offline profiling for unseen services should be clarified.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pm9T (W2–W4 & Q1–Q2)

Response to W2 & Q2: Response Fidelity and Evaluation Scope

We thank the reviewer for this valuable comment. We agree that task success and human judgments provide complementary assessments of response quality.

We would like to clarify that SLMCoach aims to preserve the original full-head model's responses while reducing LM-head memory and data movement, rather than improve the underlying model's task-solving ability. Accordingly, we evaluate response fidelity relative to full-head inference, including semantic similarity (BERTScore), lexical overlap (ROUGE, BLEU, METEOR), and generation-trajectory agreement (Sec. 4.1, Table 2). This evaluation directly measures how closely SLMCoach preserves the original model's responses under vocabulary restriction.

We acknowledge that task-level quality and human judgments are complementary evaluation dimensions, and that our current findings are limited to greedy decoding and do not establish fidelity under stochastic decoding. We will clarify these distinctions and refine the scope of our claims in the revised manuscript.


Response to W3 & Q1: Mandatory-Row Budget Enforcement

We thank the reviewer for highlighting this boundary case. Our GPU runner enforces ∣Ckreq∣≤M|C_k^{\mathrm{req}}| \le M before row selection. If ∣Ckreq∣>M|C_k^{\mathrm{req}}| > M, the input is rejected as infeasible rather than exceeding the cache budget. When ∣Ckreq∣=M|C_k^{\mathrm{req}}| = M, no additional rows are selected (Ck=CkreqC_k=C_k^{\mathrm{req}}).

We further audited all 3,842 evaluation turns and confirmed zero overflow cases (0/3,842 per configuration) across all six reported configurations, covering both KV-cached and prompt re-rendering serving. Thus, our reported results are unaffected.

We will explicitly document this feasibility check and boundary-case handling in Sec. 3.3 of the revised manuscript.


Response to W4: Runtime and Metadata Overhead

We thank the reviewer for this thoughtful observation. We agree that SLMCoach achieves comparable end-to-end latency to VocabTailor. Our primary objective is memory- and transfer-efficient inference while preserving response fidelity, rather than substantial end-to-end acceleration.

As explained in Appendix A, LM-head projection accounts for only ~3% of decoding time, limiting end-to-end speedup under Amdahl's law. Nevertheless, SLMCoach reduces LM-head memory by 99.2% and peak GPU memory by 51.5% relative to full-head inference.

To quantify the auxiliary costs, our additional profiling shows 0.738 ms/query selection time (~0.36% of E2E latency), with 5.833 MiB of retained CPU metadata and at most 0.002 MiB of GPU metadata. The estimated selection-inclusive latency remains comparable to VocabTailor (203.559 vs. 203.211 ms/query).

Importantly, the additional policy cost supports more effective cross-turn reuse: Table 5 shows that removing residency awareness increases row transfers by 19.0% (M=256), with negligible fidelity change.

AuthorsOpenReview ↗

Response to Reviewer pm9T (W1 & Q3)

Response to W1 & Q3: Generalizability and Model Selection

We thank the reviewer for highlighting these important considerations. To assess generalizability, we additionally evaluate SLMCoach on Schema-Guided Dialogue (SGD), a distinct multi-domain, multi-turn virtual assistant benchmark. Notably, all three evaluated services are unseen during offline profiling, providing a meaningful test beyond MultiWOZ 2.2.

Additional Results: SGD (KV-cached trace replay)

BudgetMethodTop-1 Coverage (%) ↑Exact Recovery (%) ↑
512VocabTailor95.6956.90
512LRU95.8656.25
512SLMCoach97.2571.88
2,048VocabTailor98.6081.64
2,048LRU97.9169.27
2,048SLMCoach99.0589.06

At M=512M=512 and M=2048M=2048, SLMCoach improves exact recovery over VocabTailor by 14.98 and 7.42 percentage points, respectively, demonstrating consistent gains across memory budgets. For unseen services, the service-specific estimate falls back to the global prior (Eq. 12), while current-query relevance and online dialogue history enable further adaptation.

We select Gemma-3-270M-IT because its compact architecture and large vocabulary make the LM-head a significant memory bottleneck (~62.6% of model weights), providing a particularly relevant testbed for memory-constrained SLM inference.

AuthorsOpenReview ↗

Response to Reviewer pm9T: Thank You for Your Positive Feedback

We sincerely thank you for your positive feedback and continued support. We truly appreciate your thoughtful comments and constructive suggestions.

We will incorporate the additional clarifications and experimental findings into the revised manuscript to further strengthen the presentation and evaluation of our work.

Thank you again for your valuable time and consideration.

Thanks for the authors' further clarifications. It is a nice work to consider LM-head cache with cross-turn locality.

Reviewer_bywLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

SLMCoach caches a subset of LM-head rows on the GPU across dialogue turns. Before each response, it ranks candidate tokens using vocabulary and service priors, associations with the current query, recent dialogue outputs, and a bonus for rows already resident on the GPU. It reserves mandatory rows, fills the remaining capacity with the highest-scoring candidates, and restricts generation to the selected vocabulary. The evaluation uses Gemma-3-270M-IT on 512 MultiWOZ 2.2 sessions containing 3,842 queries. Experiments cover both KV-cached serving and prompt re-rendering, comparing memory use, transfers, latency and output fidelity against full-head inference, Static VP, VocabTailor and LRU.

Critical strengths
  1. The paper clearly formulates LM-head row selection and caching under a GPU memory budget. Reusing rows across dialogue turns is a reasonable way to reduce repeated transfers for models with a large vocabulary-dependent memory footprint.

  2. The selection rule combines the current query with recent responses. Query-to-output associations allow it to consider tokens beyond those already present in the input, while the residency bonus accounts for the cost of loading additional rows.

Critical weaknesses
  1. Section 3.3 does not specify what happens when the mandatory rows exceed the budget. The remaining capacity is defined as (b_k=M-|C_k^{req}|), but the subsequent selection rule is undefined when this quantity is negative. The implementation needs an explicit policy for this case, along with its frequency in the evaluation.

  2. The method includes a current-query context term, and Table 5 shows that it helps. However, the experiments do not establish how well the history signal handles topic switches or unrelated follow-up queries. Recent outputs may then be a poor guide to the required vocabulary. An analysis of these cases would be more informative than the aggregate dialogue results alone.

  3. The evaluation uses one dataset and one model, so the scope of the conclusions is narrow. It also leaves room for a closer comparison with other vocabulary-selection strategies. FR-Spec and DynaSpec are relevant, although their original speculative-decoding setups use full-vocabulary verification. Adapting their selectors would require matching the row budget and accounting for training, selection and transfer costs; adding an LRU cache alone would not establish a fair comparison.

Questions for authors
  1. How is (|C_k^{req}|>M) handled, and how often does this occur under each serving mode?

  2. How does performance change when the current query is weakly related to the dialogue history? A breakdown by topic change or query-history similarity would help assess whether the history signal remains useful.

  3. Is there evidence from another dialogue dataset or SLM architecture that the observed gains extend beyond the current setting?

  4. Could the selection strategies from FR-Spec or DynaSpec be adapted to the same row budget and caching setup? Please clarify the adaptation and its costs, or explain why the comparison would not be meaningful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bywL (W1–W3 & Q1–Q2, Q4)

Response to W1 & Q1: Mandatory-Row Budget Enforcement

We thank the reviewer for identifying this missing boundary case. Our GPU runner enforces ∣Ckreq∣≤M|C_k^{\mathrm{req}}| \le M before row selection. If ∣Ckreq∣>M|C_k^{\mathrm{req}}| > M, the input is rejected as infeasible rather than exceeding the cache budget. When ∣Ckreq∣=M|C_k^{\mathrm{req}}| = M, no additional rows are selected (Ck=CkreqC_k=C_k^{\mathrm{req}}).

We audited all 3,842 evaluation turns and confirmed zero overflow cases in each of the six evaluated configurations (0/3,842 per configuration): KV-cached (M=256,512,1024,2048M=256,512,1024,2048) and prompt re-rendering (M=512,2048M=512,2048). Thus, the reported results are unaffected by this boundary case.

We will explicitly document this feasibility check and boundary-case handling in Sec. 3.3 of the revised manuscript.


Response to W2 & Q2: Topic Switching and Contextual Continuity

We thank the reviewer for this valuable suggestion. SLMCoach primarily targets topically coherent multi-turn sessions, exploiting contextual continuity for cross-turn reuse (Sec. 1, Fig. 2), rather than explicitly handling abrupt topic switches. Importantly, its current-query relevance term (Eqs. 15, 19) also adapts vocabulary selection beyond dialogue history.

Since topical continuity does not necessarily imply high lexical overlap, we conducted a query–history similarity analysis on 3,317 follow-up turns from 512 evaluation sessions, using TF–IDF similarity to the preceding eight query–response pairs. IDF weights and grouping thresholds were determined exclusively from the profiling split.

Trace-based exact recovery (%) under KV-cached serving (M=512M=512):

SimilarityVocabTailorSLMCoach (w/o Session)SLMCoach (w/o Context)SLMCoach
Low63.5565.4572.8173.85
Medium63.2963.3975.9376.23
High62.0662.7575.8876.14

Even in the Low group (zero lexical overlap), SLMCoach outperforms VocabTailor by 10.30 pp. The session estimate contributes 8.40 pp (paired session-bootstrap 95% CI: [6.63, 10.20] pp), compared with 13.39 pp in the High group. The current-query term further contributes 1.04 pp in the Low group.

These results suggest that historical signals remain beneficial under weak lexical similarity, although their contribution is smaller. We acknowledge that lexical similarity is only a proxy for topical continuity and does not establish robustness to abrupt topic switches.


Response to W3 (Baseline Comparison) & Q4: FR-Spec and DynaSpec

We thank the reviewer for suggesting these relevant methods and emphasizing fair comparison.

FR-Spec and DynaSpec optimize draft-vocabulary selection for speculative decoding, relying on full-vocabulary verification. In contrast, SLMCoach targets cross-turn LM-head caching under a strict GPU memory budget without full-vocabulary verification (Eq. 8). While their vocabulary-selection mechanisms are related, their optimization objectives and serving constraints differ substantially.

We agree that their selectors could be adapted. FR-Spec's frequency-based selection is conceptually related to our Static VP baseline, whereas adapting DynaSpec's trained, per-step router would require substantial modifications and additional training, selection, and transfer-cost accounting. Such adaptations would constitute different caching variants rather than direct comparisons with the original methods.

SLMCoach's distinct contribution is cross-turn residency-aware caching. Table 5 shows that removing the residency term increases transferred rows by 19.0% at M=256M=256, with negligible fidelity change. We will clarify these methodological distinctions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer bywL: W3 (Generalization) & Q3 (Model Selection)

Response to W3 & Q3: Generalization and Model Choice

We thank the reviewer for highlighting the importance of broader validation.

Generalization beyond MultiWOZ 2.2. To assess generalizability, we additionally evaluate SLMCoach on Schema-Guided Dialogue (SGD), a distinct multi-domain, multi-turn virtual assistant benchmark. We use 128 test sessions (768 queries) with separate training-data profiling. Notably, all three evaluated services are unseen during offline profiling, providing a meaningful test of the statistics-based selection policy beyond the original dataset and profiled services.

Additional Results: SGD (KV-cached trace replay)

BudgetMethodTop-1 Coverage (%) ↑Exact Recovery (%) ↑
512VocabTailor95.6956.90
512LRU95.8656.25
512SLMCoach97.2571.88
2,048VocabTailor98.6081.64
2,048LRU97.9169.27
2,048SLMCoach99.0589.06

At M=512M=512, SLMCoach improves exact recovery over VocabTailor by 14.98 percentage points, with a further 7.42-point gain at M=2048M=2048. These consistent improvements across two independent dialogue benchmarks, including unseen services, support the generalizability of cross-turn vocabulary reuse beyond MultiWOZ 2.2.

Rationale for model selection. We select Gemma-3-270M-IT because it provides a particularly relevant testbed for memory-constrained SLM inference. Despite its compact size and instruction-following capability, its 262K-token vocabulary results in a 320 MiB LM-head, accounting for approximately 62.6% of model weight memory (Fig. 1, Appendix A). This pronounced vocabulary-dependent bottleneck makes it especially suitable for evaluating LM-head caching under tight memory budgets. Moreover, its instruction-tuned nature enables evaluation in realistic multi-turn dialogue settings.

While cross-architecture validation remains an important direction, the additional SGD results demonstrate that SLMCoach's benefits are not confined to a single dialogue dataset or previously profiled services. We will clarify the rationale for model selection and the scope of our conclusions in the revised manuscript.

Thanks for your response. The new experiment results look impressive. The budget clarification resolves my concern about mandatory rows. And it is also shown that dialogue history still helps even when there’s little word overlap. The SGD results makes the biggest difference for me. Therefore, I will raise my score.

AuthorsOpenReview ↗

Response to Reviewer bywL: Thank You for Your Positive Feedback

We sincerely thank the reviewer for your encouraging feedback and for reconsidering your assessment. We truly appreciate your thoughtful comments and recognition of our additional experimental results.

Your constructive suggestions have been very helpful in strengthening our work. We will incorporate the clarifications and additional findings into the revised manuscript.

Thank you again for your valuable time and consideration.

Reviewer_JjchOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SLMCoach, a context-adaptive LM-head caching framework for memory-constrained multi-turn SLM inference. It uses statistical heuristics to select and retain LM-head rows across dialogue turns, reducing CPU-to-GPU data movement. Experiments on MultiWOZ 2.2, a task-oriented multi-turn dialogue benchmark, evaluate GPU memory usage, LM-head data movement, and response fidelity. Under KV-cached serving with tight cache budgets, SLMCoach achieves higher exact recovery than VocabTailor and LRU.

Critical strengths

The paper addresses cross-turn LM-head reuse in multi-turn inference, a relevant problem for memory-constrained virtual assistants. Its cache-management policy considers both token demand and transfer cost through token prevalence, current-query relevance, and GPU residency.

Critical weaknesses

The applicability of the statistics-based selection policy beyond MultiWOZ 2.2 remains unclear. The policy relies on empirical token statistics from profiling data, but the experiments cover only this dataset. The paper does not report results on broader question-answering or reasoning benchmarks with task-specific metrics.

Questions for authors

Can SLMCoach be extended to broader virtual assistant tasks? What profiling data would be needed, and how would the method handle out-of-distribution queries or differences in users’ vocabulary?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Jjch (W1 & Q1)

Response to W1: Generalization Beyond MultiWOZ 2.2

We thank the reviewer for highlighting this important concern. To assess generalizability, we additionally evaluate SLMCoach on Schema-Guided Dialogue (SGD), a distinct multi-domain, multi-turn virtual assistant benchmark. Notably, all three evaluated services are unseen during offline profiling, providing a meaningful test of our statistics-based selection policy beyond MultiWOZ 2.2.

Additional Results: SGD (KV-cached trace replay)

BudgetMethodTop-1 Coverage (%) ↑Exact Recovery (%) ↑
512VocabTailor95.6956.90
512LRU95.8656.25
512SLMCoach97.2571.88
2,048VocabTailor98.6081.64
2,048LRU97.9169.27
2,048SLMCoach99.0589.06

128 SGD test sessions (768 queries) from one official test shard; separate SGD training data for profiling; fixed full-head reference trajectories.

At M = 512, SLMCoach improves exact recovery over VocabTailor by approximately 15 percentage points (paired session-bootstrap 95% CI: [12.03, 17.93]), with a 7.42 pp gain at M = 2,048. These results show that global vocabulary priors and online contextual adaptation enable effective selection even for unseen services, supporting generalization beyond MultiWOZ 2.2.

We appreciate the reviewer's suggestion and agree that task-specific metrics provide complementary evidence. SLMCoach focuses on latency-sensitive, memory-constrained multi-turn inference, aiming to preserve response fidelity relative to full-head inference rather than improve the underlying model's task-solving ability. Our original semantic-similarity and generation-agreement evaluations (Sec. 4.1, Table 2) are complemented by the new SGD reference-token coverage results.


Response to Q1: Broader Virtual Assistant Tasks and Vocabulary Adaptation

We thank the reviewer for these thoughtful questions. SLMCoach is designed for multi-turn virtual assistant tasks with contextual continuity, rather than being restricted to a particular dialogue benchmark. Offline profiling uses representative queries and full-head model responses to estimate global token prevalence and query–output associations, without requiring service- or user-specific profiles.

For out-of-distribution queries and differences in users' vocabulary, SLMCoach adapts through three complementary mechanisms: (1) global-prior fallback for unseen services (Eq. 12), (2) current-query relevance and mandatory query-token retention to accommodate new inputs (Eqs. 15, 20), and (3) recent dialogue prevalence and cross-turn reuse to capture evolving vocabulary usage (Eqs. 13, 16–19).

Our additional SGD results (W1) provide empirical evidence of this adaptability, showing approximately 15 pp higher exact recovery over VocabTailor on unseen services at M=512M=512. Furthermore, our query–history analysis (Response to bywL, W2) shows a 10.30 pp improvement in exact recovery over VocabTailor, even when the current query has zero lexical overlap with recent dialogue history. Together, these results highlight SLMCoach's ability to combine offline vocabulary knowledge with online contextual adaptation, supporting its applicability across different services and dialogue contexts.

Thanks to the authors for the response and additional experiments. I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer Jjch (Q1 Revised): Broader Virtual Assistant Tasks and Vocabulary Adaptation

Response to Q1: Broader Virtual Assistant Tasks and Vocabulary Adaptation

We thank the reviewer for these thoughtful questions. SLMCoach is designed for multi-turn virtual assistant tasks with contextual continuity, rather than being restricted to a particular dialogue benchmark. Its selection policy relies on token-level statistics and online dialogue signals rather than dataset-specific rules, enabling adaptation across different dialogue settings without modifying the underlying framework.

Offline profiling requires representative queries and full-head model responses to estimate global token prevalence and query–output associations, without requiring user-specific profiles or prior observations of every service. For unseen services and out-of-distribution queries, SLMCoach combines three complementary mechanisms: (1) global-prior fallback when service-specific statistics are unavailable (Eq. 12), (2) current-query relevance and mandatory query-token retention to accommodate new inputs (Eqs. 15, 20), and (3) recent dialogue prevalence and cross-turn reuse to adapt to evolving vocabulary usage (Eqs. 13, 16–19). These mechanisms enable vocabulary adaptation without requiring prior profiling of every encountered service.

Our additional SGD results (W1) provide empirical evidence beyond MultiWOZ 2.2, showing 14.98 pp higher exact recovery over VocabTailor at M=512M=512, with all three evaluated services unseen during offline profiling.

To further examine adaptability under weak query–history similarity, we analyze 3,317 follow-up turns from MultiWOZ 2.2 using TF–IDF similarity to the preceding eight query–response pairs (Response to bywL, W2). IDF weights and grouping thresholds are determined exclusively from the profiling split.

Query–History Similarity Analysis on MultiWOZ 2.2: Trace-based Exact Recovery (%) under KV-cached serving (M=512M=512)

SimilarityVocabTailorSLMCoach (w/o Session)SLMCoach (w/o Context)SLMCoach
Low63.5565.4572.8173.85
Medium63.2963.3975.9376.23
High62.0662.7575.8876.14

Even in the Low group (zero lexical overlap), SLMCoach outperforms VocabTailor by 10.30 pp. The session estimate contributes 8.40 pp, while the current-query context term further contributes 1.04 pp, demonstrating their complementary benefits under weak lexical similarity.

Together, the SGD and MultiWOZ 2.2 query–history analyses provide complementary evidence of SLMCoach's adaptability across unseen services and varying degrees of contextual similarity. These results highlight the benefit of combining global vocabulary priors, current-query relevance, and online dialogue adaptation for broader multi-turn virtual assistant settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.