- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes storing transformer weight matrices as learned latent codes and small per-matrix generators. A generator expands each code into 16 weights distributed across a global grid of matrix tiles. Generated matrices can be discarded and recomputed, reducing the number of parameters requiring gradients and optimizer states. The resulting Bicameral models retain the dense transformer's projection dimensions and dense multiplication cost, while storing fewer independent parameters.
Critical strengths
- The main experimental result and ablations show the feasibility of the proposed method. The six downstream tasks provide useful corroboration beyond the training objective.
- The study reports relevant costs as well as savings. Figure 4 distinguish training with and without checkpointing and inference with generic versus optimized generation. Authors explicitly acknowledge unchanged dense-model FLOPs, 4-6% overhead at batch 8, dependence on activation memory, and untested extrapolation beyond 2B effective parameters and 10B tokens.
Critical weaknesses
- There is a lack of comparisons with other memory-efficient pre-training baselines under matched budgets, e.g. GaLore, APOLLO or FRUGAL [1,2,3]. It would be also useful to compare Bicameral with Monarch [4] algorithm in the same setting and identify the differences with linear generators. Tables 2-3 show, that the standard models use the base learning rate for embeddings and normalization gains, whereas Bicameral uses twice that rate. The methods also differ in initialization scale, code/generator learning rates, and weight decay, which raises the issue about effective tuning for competitors.
[1] Zhao J. et al. Galore: Memory-efficient llm training by gradient low-rank projection //arXiv preprint arXiv:2403.03507. – 2024.
[2] Zhu H. et al. Apollo: Sgd-like memory, adamw-level performance //Proceedings of Machine Learning and Systems. – 2025. – Т. 7.
[3] Zmushko P. et al. FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training //arXiv preprint arXiv:2411.07837. – 2024.
[4] Dao T. et al. Monarch: Expressive structured matrices for efficient and accurate training //International Conference on Machine Learning. – PMLR, 2022. – С. 4690-4721.
Questions for authors
- Does a trained linear generator with the same global layout recover most of the reported quality?
- How do fine-tuned dense and low-rank baselines perform when embedding/norm learning-rate multipliers and initialization are included in their search?
4 comments
References
[1] Biderman et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. ICML.
[2] Dao et al. (2022). Monarch: Expressive Structured Matrices for Efficient and Accurate Training. ICML.
[3] Qiu et al. (2024). Compute Better Spent: Replacing Dense Layers with Structured Matrices. ICML.
[4] Touvron et al. (2023a). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
[5] Touvron et al. (2023b). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
[6] Zhao et al. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML.
[7] Zhu et al. (2025). APOLLO: SGD-like Memory, AdamW-level Performance. MLSys.
[8] Zmushko et al. (2024). FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training. arXiv:2411.07837.
Thank you again for the time and effort you have devoted to reviewing our work. As the discussion period is drawing to a close, we wanted to kindly follow up to see if our response and additional experiments have addressed your questions and concerns. We would greatly appreciate your feedback and are more than happy to clarify anything that remains open.
I thank authors for a rigorous ablation studies that they provided. I assume, that the vast majority of the questions are covered, however, there are still some concerns about the non-linear generators: they still are not this profitable compared to the linear ones, as the study shows, and the benefits are negligible. However, I appreciate the thorough work done by authors during the discussion period and will raise my score to 3.
We thank the reviewer for the thorough review of our experimental controls, ablations, and related work. We provide comparisons with GaLore, APOLLO, and Monarch below. We further discuss hyperparameter selection, linear vs. nonlinear generators, and baseline tuning.
Comparison with GaLore, APOLLO
These memory-efficient pretraining methods only compress the optimizer states [6][7][8]. The model's weight memory usage remains uncompressed, whereas Bicameral reduces all of the above by directly reducing trainable parameter count. The two are not mutually exclusive: these methods can be applied to Bicameral to reduce training memory further.
Comparison with Monarch
In Dao et al.[2], Monarch pre-training switches to dense weights before the end of training, and thus does not preserve its memory savings throughout. For a matched-budget comparison, we trained Monarch compressed end-to-end, matching Bicameral's exact stored parameter counts and training state.
All added baselines use warm-up settings from their respective papers and the best point of a LR sweep, trained to 1B tokens.
Hyperparameter selection
Q1. Generator linearity
We trained smaller proxy models from scratch with the same global layout. A linear generator recovers most of the quality.
Separately, trained 2B models do utilize the nonlinearity. Below we replaced each trained generator with its best linear or quadratic fit, without retraining. We found that the trained models depend on the nonlinearity mainly in attention:
Q2. Baselines optimization