- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SLMCoach, a context-adaptive LM-head cache for multi-turn small language model inference. It combines offline vocabulary profiling, dialogue-dependent relevance and prevalence, and residency-aware selection to retain useful output rows on the GPU and transfer only missing rows between turns. Experiments on MultiWOZ with Gemma-3-270M-IT report substantial memory and transfer reductions, with broadly similar outputs to full-head inference. Comparisons across cache budgets and serving modes, together with component ablations, indicate that explicit reuse is most beneficial under KV-cached serving and tight budgets.
Critical strengths
- The paper targets a practical, underexplored source of inference overhead by exploiting cross-turn locality in vocabulary-dependent weights.
- The framework coherently combines dialogue relevance, recent token usage, and cache residency to balance output coverage against parameter transfers.
- The experiments demonstrate substantial LM-head memory and transfer savings while maintaining broadly comparable response fidelity to full-head inference.
Critical weaknesses
-
Evaluation is confined to a narrowly scoped dialogue benchmark, a compact model, and desktop GPU hardware, leaving broader model and edge-deployment applicability insufficiently established.
-
Quality evaluation measures agreement with full-head outputs under greedy decoding rather than task success or human judgments, limiting support for the claim of preserved response quality.
-
The cache policy requires all serving-context token rows to be mandatory but does not explain how the strict memory budget is maintained when this set exceeds capacity.
-
End-to-end latency is essentially unchanged against the strongest query-isolated baseline, while selection overhead and auxiliary metadata memory are not separately quantified.
Questions for authors
- How is the cache budget enforced when mandatory serving-context rows exceed capacity, and does the evaluation cover such cases under either serving mode?
- Task-oriented quality or human assessment alongside full-head agreement should be reported, and whether the fidelity conclusions extend beyond greedy decoding?
- The memory–quality trade-off should be validated on another model or in a less constrained dialogue setting, and the behavior of offline profiling for unseen services should be clarified.
4 comments
Response to Reviewer pm9T (W1 & Q3)
Response to W1 & Q3: Generalizability and Model Selection
We thank the reviewer for highlighting these important considerations. To assess generalizability, we additionally evaluate SLMCoach on Schema-Guided Dialogue (SGD), a distinct multi-domain, multi-turn virtual assistant benchmark. Notably, all three evaluated services are unseen during offline profiling, providing a meaningful test beyond MultiWOZ 2.2.
Additional Results: SGD (KV-cached trace replay)
| Budget | Method | Top-1 Coverage (%) ↑ | Exact Recovery (%) ↑ |
|---|---|---|---|
| 512 | VocabTailor | 95.69 | 56.90 |
| 512 | LRU | 95.86 | 56.25 |
| 512 | SLMCoach | 97.25 | 71.88 |
| 2,048 | VocabTailor | 98.60 | 81.64 |
| 2,048 | LRU | 97.91 | 69.27 |
| 2,048 | SLMCoach | 99.05 | 89.06 |
At and , SLMCoach improves exact recovery over VocabTailor by 14.98 and 7.42 percentage points, respectively, demonstrating consistent gains across memory budgets. For unseen services, the service-specific estimate falls back to the global prior (Eq. 12), while current-query relevance and online dialogue history enable further adaptation.
We select Gemma-3-270M-IT because its compact architecture and large vocabulary make the LM-head a significant memory bottleneck (~62.6% of model weights), providing a particularly relevant testbed for memory-constrained SLM inference.
Response to Reviewer pm9T: Thank You for Your Positive Feedback
We sincerely thank you for your positive feedback and continued support. We truly appreciate your thoughtful comments and constructive suggestions.
We will incorporate the additional clarifications and experimental findings into the revised manuscript to further strengthen the presentation and evaluation of our work.
Thank you again for your valuable time and consideration.
Thanks for the authors' further clarifications. It is a nice work to consider LM-head cache with cross-turn locality.
Response to Reviewer pm9T (W2–W4 & Q1–Q2)
Response to W2 & Q2: Response Fidelity and Evaluation Scope
We thank the reviewer for this valuable comment. We agree that task success and human judgments provide complementary assessments of response quality.
We would like to clarify that SLMCoach aims to preserve the original full-head model's responses while reducing LM-head memory and data movement, rather than improve the underlying model's task-solving ability. Accordingly, we evaluate response fidelity relative to full-head inference, including semantic similarity (BERTScore), lexical overlap (ROUGE, BLEU, METEOR), and generation-trajectory agreement (Sec. 4.1, Table 2). This evaluation directly measures how closely SLMCoach preserves the original model's responses under vocabulary restriction.
We acknowledge that task-level quality and human judgments are complementary evaluation dimensions, and that our current findings are limited to greedy decoding and do not establish fidelity under stochastic decoding. We will clarify these distinctions and refine the scope of our claims in the revised manuscript.
Response to W3 & Q1: Mandatory-Row Budget Enforcement
We thank the reviewer for highlighting this boundary case. Our GPU runner enforces ∣Ckreq∣≤M before row selection. If ∣Ckreq∣>M, the input is rejected as infeasible rather than exceeding the cache budget. When ∣Ckreq∣=M, no additional rows are selected (Ck=Ckreq).
We further audited all 3,842 evaluation turns and confirmed zero overflow cases (0/3,842 per configuration) across all six reported configurations, covering both KV-cached and prompt re-rendering serving. Thus, our reported results are unaffected.
We will explicitly document this feasibility check and boundary-case handling in Sec. 3.3 of the revised manuscript.
Response to W4: Runtime and Metadata Overhead
We thank the reviewer for this thoughtful observation. We agree that SLMCoach achieves comparable end-to-end latency to VocabTailor. Our primary objective is memory- and transfer-efficient inference while preserving response fidelity, rather than substantial end-to-end acceleration.
As explained in Appendix A, LM-head projection accounts for only ~3% of decoding time, limiting end-to-end speedup under Amdahl's law. Nevertheless, SLMCoach reduces LM-head memory by 99.2% and peak GPU memory by 51.5% relative to full-head inference.
To quantify the auxiliary costs, our additional profiling shows 0.738 ms/query selection time (~0.36% of E2E latency), with 5.833 MiB of retained CPU metadata and at most 0.002 MiB of GPU metadata. The estimated selection-inclusive latency remains comparable to VocabTailor (203.559 vs. 203.211 ms/query).
Importantly, the additional policy cost supports more effective cross-turn reuse: Table 5 shows that removing residency awareness increases row transfers by 19.0% (M=256), with negligible fidelity change.