acceptodds
Under review as a conference paper at ICLR 2027

OrBIT: Structure-Guided Embedding Compression

Abstract

Embedding tables are among the largest components of modern language models. Most compression methods fix a coding geometry such as coordinate blocks, low-rank subspaces, or unrestricted codebooks, and optimize within it. We instead ask whether the coding geometry can itself be discovered. We introduce OrBIT, a structure-guided embedding compression framework that learns reusable local geometry from orbit dynamics and uses it to constrain a small set of shared codewords. The global reconstruction residual then decides where the fixed coding budget is spent, while redundant overlapping charts let local errors compensate one another after gluing. Our theory shows how tight-chart geometry controls distortion, how the global residual directs sequential allocation, and how data-geometry-guided refinement improves the codec. The resulting orbit machinery is compiled away, leaving a compact decoder in which the learned structure governs what is stored, where capacity is allocated, and how local information is assembled globally. Across four LLM embedding tables, OrBIT achieves compression on GPT-2 and over on each 7B table relative to 16-bit storage, while delivering competitive rate-distortion performance against established quantization and low-rank baselines.

Primary area:
learning theory
Keywords:
learned geometry, structured codebooks, orbit dynamics, representation compression, local-to-global learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_a1xJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an embedding-table compression framework named OrBIT. It uses data-derived orbit scaffolds to structure shared additive codebooks. Among multiple backbone models, the proposed OrBIT achieves better compression while delivering competitive rate-distortion performance against baselines.

Critical strengths

The paper proposes a new compression framework to reduce embedding-table storage through orbit-guided shared codebooks and overlapping local representations.

Critical weaknesses
  • The novelty of the paper is limited and missing the comparison with SOTA baselines. The major contribution of this paper is orbit-guided codebook construction, but the experiments do not convincingly demonstrate its gains. In the main experiments, the learned orbit scaffolds are full rank, meaning they do not restrict the codewords to a lower-dimensional subspace. Moreover, the authors compare orbit scaffolds only against random Gaussian subspaces rather than stronger data-driven alternatives such as PCA or conventional dictionary learning. Some SOTA work are not compared in the paper such as CARVQ (EMNLP 2025) and QINCo2 (ICLR2025).

  • The motivation of this paper is unclear. The embedding table is relatively small part of LLM and the compression on the embedding table is quite limited on the overall memory saving. Taking the example of Llama-7B with 32K 4096 embedding table, the embedding table is only about 1.9% of the total model parameters. However, it introduces a significant compression/decompression overhead. The authors should provide the overall memory saving based on the proposed scheme.

Questions for authors

See the weakness above.

Reviewer_DPG3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

Proposes OrBIT, a framework compressing LM embedding tables using shared additive codebooks and orbit-based scaffolds. It utilizes tight-frame charts, residual-guided scheduling, and post-gluing refinement. The orbit construction is discarded post-compilation, yielding a compact decoder. Theoretical analyses cover reconstruction errors and refinement. Experiments on GPT-2 and 7B models demonstrate high compression (up to 37.9× vs FP16) and competitive rate-distortion against PQ, OPQ, and AQ baselines.

Critical strengths
  1. Principled Formulation: Integrates additive coding with tight-frames, supported by clear theoretical error decomposition and refinement conditions.
  2. Deployable Decoder: Explicitly accounts for all storage components, verifying that orbit-construction overhead can be fully removed post-compilation.
  3. Comprehensive Evaluation: Conducts extensive mechanism-oriented diagnostics and ablations, showing post-gluing refinement consistently reduces error (2.8–4.1%) without extra storage.
Critical weaknesses
  1. Unclear Orbit Contribution: Accepted scaffolds have full chart rank, imposing no effective subspace restriction. The specific benefit of orbit dynamics over simple anchor-based construction remains unclear.
  2. Weak Geometry Baselines: Only compares against random Gaussian subspaces, lacking comparison with simpler data-aware alternatives like rank-matched PCA or plain anchor spans.
  3. High Distortion & Limited Validation: 7B models show high end-to-end reconstruction errors (~0.92) and large PCA bitrate overhead (92%). Evaluations only use 10k-token subsets and lack LM perplexity or downstream task metrics.
  4. Inconsistent Baseline Comparisons: Gains over PQ and OPQ are modest and inconsistent (e.g., PQ-q8 outperforms OrBIT on Mistral-7B-Instruct). Comparisons with modern methods like QINCo are missing.
  5. Unverified Efficiency & Novelty: Theoretical contributions appear incremental. Lacks quantitative evaluation of scheduling benefits, encoder costs, and decoding throughput.
Questions for authors
  1. Orbit Necessity: Is orbit construction essential given full-rank scaffolds? Compare against direct optimization, sparse realization, or random dictionaries to isolate the benefit of wrapped-shift dynamics.
  2. Stronger Baselines: How does orbit geometry compare to rank-matched PCA or plain anchor-span baselines? What is the specific advantage of orbit learning over simply selecting representative anchors?
  3. Downstream Performance: Provide perplexity or downstream task metrics on full vocabulary tables. Quantify the PCA-only reconstruction floor and evaluate competitiveness at larger dimensions with lower distortion.
  4. Ablations & Efficiency: Provide matched-budget ablations for scheduling, orbit dictionaries, and refinement. Report encoder runtime, peak memory, and decoder throughput to establish practical advantages.
Reviewer_r6xSOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OrBIT proposes a structure-guided compression framework for LLM embedding tables, motivated by their large memory footprint. OrBIT relies on the following main ideas:

Overlapping charts: The embedding space is viewed through L overlapping rotated coordinate windows called charts.

Orbit scaffolds: In each chart, a small set of achor tokens generates a dictionary of atoms by repeatedly applying a learned operator. The span of these atoms is the scaffold.

Shared addtive codebooks: Each token is stored as TT small indices per chart, and each index selects a shared codeword. The codewords are realized as sparse combinations of atoms and quantized to 8 bits.

Residual-driven allocation: Stages are built greedily. Each new stage goes to the chart that currently has the highest reconstruction error. ℓs∈\mboxargmaxℓ∈Es∥RℓG(s−1)∥μ\ell_s \in \mbox{arg max}_{\ell \in \Epsilon_s} \| R_{\ell} G^{(s-1)}\|_\mu, where G(s−1)=E−E^(s−1)G^{(s-1)} = E - \hat{E}^{(s-1)} is the global residual.

Gluing and compilation: The local reconstructions are combined by ``tight-frame synthesis to couple local shared-codebook representations''. Since the charts overlap, the inconsistent local errors are partially canceled. The orbit machinery is later discarded and the decoder keeps only the PCA map, the charts, the quantized codewords and the token indices.

Critical strengths

Overlapping tight frame plus scheduling and refinement fitted to the global residual, applied to additive coding in a PCA subspace.

Theorem 3.3 gives an exact error for canonical tight-frame gluing ∥E−E^∥μ=1A(ε\mboxloc2(E)−ε\mboxinc2(E))12\|E - \hat{E} \|_\mu = \frac 1 {\sqrt{A} } \left( \varepsilon_{\mbox{loc}}^2(E) - \varepsilon_{\mbox{inc}}^2(E) \right)^{\frac 1 2}

Critical weaknesses

On the Experiments section, for GPT-2 the size of the token table is D=10,000D = 10,000. The orbits scafolds are constructed from0.75%from 0.75\% of the token table using charts of dimension Mℓ=32M_\ell = 32. There are more anchors than dimensions. So, Sℓ:=spanDℓ=ZℓS_\ell := \text{span} \mathcal{D}_\ell = \mathcal{Z}_\ell, and the acceptance test equation (4) εproj,ℓ:=∥(I−ΠSℓ)Z(ℓ)∥μ≤τproj\varepsilon_{\text{proj}, \ell} := \| (I - \Pi_{S_\ell}) Z^{(\ell)} \|_\mu \leq \tau_{\text{proj}} is trivially satisfied. I.e. the scaffold does does not impose any constraint on the codewords.

Questions for authors

What is your definition of coding geometry? The term is used through out the document; but, the reviewer did not find an explicit definition. Does it mean the set of all reconstructions that the decoder family can produce?

Reviewer_PVFdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper introduces OrBIT, an embedding compression framework that combines anchor-generated orbit dictionaries with shared additive codebooks on overlapping local charts. Global reconstruction residuals guide codebook construction and refinement, while the orbit machinery is discarded after compilation into a compact decoder. The authors provide theoretical analyses of reconstruction error and refinement, alongside explicit storage accounting. Experiments on sampled embedding tables from four language models report competitive rate–distortion performance against quantization baselines, with ablations examining scaffold alignment, refinement, redundancy, and decoder compilation.

Critical strengths

(1) The paper accounts for decoder storage, including the PCA representation, chart descriptions, codebooks, scales, and token indices. Comparisons with baselines using matched codeword precision make the reported compression results more transparent.

(2) The theoretical analysis connects local approximation errors to global reconstruction and derives explicit conditions for refinement to reduce distortion. This provides an explanation of how overlapping representations interact during reconstruction.

Critical weaknesses

(1) Evaluation is limited to sampled tables of 10,000 embeddings, without full-vocabulary compression or language-model evaluation. The 7B results have relative reconstruction errors around 0.92 and mean cosine similarities around 0.38–0.40. Without perplexity or downstream results, the reported compression ratios provide limited evidence of useful model compression.

(2) Improvements over PQ and OPQ are small at the representative operating points, and OrBIT performs worse on Mistral-7B-Instruct. Although comparisons against AQ are more favorable, they do not identify which components account for the advantage.

(3) Numerous definitions and propositions precede an algorithmic overview, while the main text lacks a pipeline diagram illustrating orbit construction, overlapping charts, and decoder compilation. The rate–distortion plots are relegated to the appendix. Repeated conceptual explanations further obscure the distinction between established ingredients and the specific contribution of orbit dynamics.

(4) Line 423 is incomplete.

Questions for authors

Please see the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.