- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an embedding-table compression framework named OrBIT. It uses data-derived orbit scaffolds to structure shared additive codebooks. Among multiple backbone models, the proposed OrBIT achieves better compression while delivering competitive rate-distortion performance against baselines.
Critical strengths
The paper proposes a new compression framework to reduce embedding-table storage through orbit-guided shared codebooks and overlapping local representations.
Critical weaknesses
-
The novelty of the paper is limited and missing the comparison with SOTA baselines. The major contribution of this paper is orbit-guided codebook construction, but the experiments do not convincingly demonstrate its gains. In the main experiments, the learned orbit scaffolds are full rank, meaning they do not restrict the codewords to a lower-dimensional subspace. Moreover, the authors compare orbit scaffolds only against random Gaussian subspaces rather than stronger data-driven alternatives such as PCA or conventional dictionary learning. Some SOTA work are not compared in the paper such as CARVQ (EMNLP 2025) and QINCo2 (ICLR2025).
-
The motivation of this paper is unclear. The embedding table is relatively small part of LLM and the compression on the embedding table is quite limited on the overall memory saving. Taking the example of Llama-7B with 32K 4096 embedding table, the embedding table is only about 1.9% of the total model parameters. However, it introduces a significant compression/decompression overhead. The authors should provide the overall memory saving based on the proposed scheme.
Questions for authors
See the weakness above.