acceptodds
Under review as a conference paper at ICLR 2027

A Distribution-Aware Approximate LM Head for Reduced-Memory Decoding

Abstract

The output head of a large language model scores the entire vocabulary at every decoding step while most decoding approaches use only a small set of candidates, but this operation can be taken as a maximum inner-product search problem in which the rows of the LM head are fixed items and each hidden state is a query. We study when approximate vocabulary search can replace the dense head without changing decoding behavior. Standard search systems however optimize top- recall, a poor target here because it does not take probabilities into consideration. We propose the \MethodFull (\method), which searches a compact graph index over the rows of the LM head, optionally reranks the returned candidates, normalizing only those. At a fixed search budget, \method's time and transient memory do not depend on vocabulary size, and three operating regimes optimize our approach for low latency, low memory, or a balance of the two. On four 7–8B LLMs, rescoring the 64 returned candidates with exact logits keeps \method within percentage points of dense multiple-choice accuracy on every language-matched benchmark, whereas decoding from its own approximate scores loses at most points but can be recovered for about of serving throughput with a low-margin rescoring fallback while reducing the head's share of resident GPU memory by at least in vLLM, where the dense head holds up to of resident GPU memory.

Primary area:
generative models
Keywords:
Approximate nearest-neighbor search, Memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KdBmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a distribution-aware approximate language-model head intended to reduce the memory and computational cost of large-vocabulary decoding. Instead of computing logits over the entire vocabulary, the method performs graph-based approximate maximum-inner-product search over compressed output embeddings, retains a shortlist of 64 tokens, and normalizes the output distribution over that shortlist. The paper further proposes Jensen–Shannon divergence to evaluate shortlist quality. Evaluation covers four 7–8B models and multiple-choice benchmarks, replay-based reasoning evaluations, limited sampling to demonstrate the performance gains.

Critical strengths

The paper provides a principled distribution-level measurement of approximate LM-head errors. Jensen–Shannon divergence is more informative than top-kk recall alone because it accounts for the probability mass of omitted tokens. The separation of retrieval error from approximate-score error, together with per-step certificates, provides a useful framework for analyzing approximate output layers.

Critical weaknesses
  1. The paper does not demonstrate a compelling end-to-end deployment point. The low-latency configuration uses more resident head memory than the dense head, while its throughput changes are modest and inconsistent. The low-memory configuration substantially reduces head memory but is slower in most serving results, with throughput losses reaching roughly 52%. The balanced configuration is not evaluated in the serving implementation. Moreover, the LM head accounts for at most 5.5% of total resident GPU memory in the reported setups, and the paper does not establish that reducing this component increases supported concurrency or context length, or lowers serving cost. The serving study is also limited to eight concurrent requests with greedy decoding. Consequently, the paper demonstrates a tradeoff between local head memory and latency, but not a convincing system-level benefit under realistic deployment constraints.
  2. The quality and generalization evidence is misaligned with the deployed method and too narrow for the scope of the claims. The operating regimes are selected using JSD after exact rescoring rather than the JSD or downstream accuracy of the approximate scores used in deployment. The strongest quality result therefore describes a rescored ceiling, whereas the deployed configurations show losses of up to 7.1% on language-matched benchmarks and substantially weaker certification and token-agreement rates. Stochastic decoding is evaluated only for the rescored arm and primarily at answer positions, while reasoning evaluations replay cached dense trajectories and therefore do not capture autoregressive error propagation. Finally, all four evaluated models are 7–8B models with the same hidden dimension, and their vocabularies range only up to approximately 197K tokens. The results therefore do not establish robustness across larger vocabularies (for example, Qwen3.8 with 256k vocabulary), model sizes, hidden dimensions, or long-horizon applications such as code generation and tool-using agents.
Questions for authors
  1. What specific reported configuration should be regarded as the intended deployment point, and what end-to-end serving benefit does it provide over the dense head once total model memory, throughput, and supported request capacity are considered together?
  2. Why were the three operating regimes selected using exactly rescored JSD rather than the deployed distribution produced by the approximate scores? Were any deployed-arm quality measurements involved in configuration selection?
  3. Does the paper contain any stochastic-decoding comparison using the deployed arm? Table 11 appears to evaluate only exact rescoring at answer positions, while the batch-2048 result appears to be a sampler microbenchmark rather than an end-to-end approximate-head measurement.
Reviewer_s2raOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes the Approximate Head (AH), which replaces the dense LM head with approximate nearest-neighbour search over compressed copies of its rows. At each decoding step, AH searches a compact index, keeps the best candidates it visits, optionally reranks them, and normalises the final scores over those candidates. The authors demonstrated that AH runs in the vLLM inference engine on five GPU types with the dense weights removed, cutting the head’s share of resident memory by at least 2.7×.

Critical strengths
  • I believe this is a timely problem, and the motivation of the study is well structured.
  • The engineering effort is considerable: a drop-in vLLM head, four models with 100k-197k vocabularies, five GPUs and multilingual benchmarks.
Critical weaknesses
  • Some of the key baselines based on a quantised dense head are missing
  • Some results may not match well.
  • Speculative decoding-related claim lacks experimental evidence.
Questions for authors

Here is my feedback on this paper below.

First of all, some of the key baselines based on a quantised dense head are missing. It seems the large portion of memory saving comes from storing rows in FP8 or 4-bit precision, so it is natural to include those as baselines. A 4-bit weight-only head (GPTQ or AWQ) would provide as much as 4x memory savings compared to FP16 precision. These methods are cited but never compared against. I would like to suggest that the authors conduct further experiments with these baselines.

Second, some result numbers may not match well. Table 2 puts AH_{l} at 109.8% of dense; however, every L40S, A100 and H100 row of Table 7 shows AH_{l} has a smaller memory share than dense (2.93% vs. 3.08% for Olmo-3-7B). Could you clarify this point?

Third, the authors mention that AH can replace the target head in speculative decoding. However, it is not supported with experimental evidence. Specifically, lossless speculative decoding requires the exact target probability of each drafted token over the full vocabulary. With a 64-token shortlist, tokens outside it are rejected outright, so the output follows AH's distribution rather than the target's. I would like to suggest that the authors provide evidence regarding this point with acceptance length and agreement for at least one drafter.

Reviewer_tQstOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To reduce GPU memory for the LM head during inference, AH (approximate head) replaces the dense LM head with a compressed copy (4/8-bit) and a nearest-neighbor graph. At each decoding step, the hidden state is used as a query for a greedy walk on that graph. The walk starts from random tokens, repeatedly expands the best-scoring nodes, and scores only the rows it visits using the compressed weights. The top 64 tokens it finds (optionally re-scored with a fallback copy) are softmaxed into the output distribution. The search settings are tuned to minimize the JS divergence from the dense head's distribution rather than top-k recall. Empirical validation shows that it can reduce 5.5% of GPU memory while keeping within 0.1pt of accuracy (if rescored using exact logits).

Critical strengths
  • The nearest-neighbor graph is clever and novel.
  • Accuracy: the method with rescoring using exact logits is close to exactly using the LM head.
Critical weaknesses
  • The absolute reduction is quite small (5.5%), since the LM head memory is typically not very large (1-2GB).
  • The near-lossless results come from the variant that needs full weights. It's just that the weights are kept in CPU memory instead of GPU memory.
  • Baseline: The paper does not compare against weight-only quantization of the LM head (e.g., INT4 W4G128 via AutoRound/GPTQModel, or Q6_K in llama.cpp), which is supported in vLLM. Public numbers from AutoRound show an average loss under 0.3 pt with a head of about 1/4 the size. That would be smaller than AH_m's 285 MiB on Olmo-3.
Questions for authors
  • What would quantization (4-8 bits) of the LM head get in terms of quality?
  • What happens if the model has tied embedding and LM head matrix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.