acceptodds
Under review as a conference paper at ICLR 2027

Bicameral: Memory-Efficient Transformer via Latent Tensor Rematerialization

Abstract

Memory capacity limits the size of language models that can be trained and served on given hardware. Quantization lowers the precision of each weight, but leaves the number of trainable parameters unchanged, and with it the gradients and optimizer states that dominate training memory. We introduce Latent Tensor Rematerialization (LTR), which generates each weight matrix on demand from compressed latent codes with small learned generators, and in doing so converts a standard matrix into a recomputable activation. This reduces memory usage by directly storing fewer parameters and optimizer states, as an alternative to lowering precision. Moreover, LTR leverages underutilized compute to reduce the number of parameters that must be stored, and its full-precision parameters remain compatible with quantization. We validate the effectiveness of LTR with Bicameral, a LLaMA-style transformer architecture whose weight matrices are generated on demand by small decoders. %When trained to 10B tokens, a Bicameral model with half the stored parameters of a 2B-parameter transformer matches its validation perplexity and outperforms a standard transformer with the same number of stored parameters. We train Bicameral models with 2B effective parameters to 10B tokens, and find that Bicameral matches the uncompressed baseline on perplexity and zero-shot benchmarks, while reducing stored parameters by 50% and training memory usage by up to 48%.

Primary area:
optimization
Keywords:
Large Language Models, Transformer Architecture, Tensor Rematerialization, Memory Efficient Training, Model Compression, Hypernetworks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qf5QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes storing transformer weight matrices as learned latent codes and small per-matrix generators. A generator expands each code into 16 weights distributed across a global grid of matrix tiles. Generated matrices can be discarded and recomputed, reducing the number of parameters requiring gradients and optimizer states. The resulting Bicameral models retain the dense transformer's projection dimensions and dense multiplication cost, while storing fewer independent parameters.

Critical strengths
  • The main experimental result and ablations show the feasibility of the proposed method. The six downstream tasks provide useful corroboration beyond the training objective.
  • The study reports relevant costs as well as savings. Figure 4 distinguish training with and without checkpointing and inference with generic versus optimized generation. Authors explicitly acknowledge unchanged dense-model FLOPs, 4-6% overhead at batch 8, dependence on activation memory, and untested extrapolation beyond 2B effective parameters and 10B tokens.
Critical weaknesses
  • There is a lack of comparisons with other memory-efficient pre-training baselines under matched budgets, e.g. GaLore, APOLLO or FRUGAL [1,2,3]. It would be also useful to compare Bicameral with Monarch [4] algorithm in the same setting and identify the differences with linear generators. Tables 2-3 show, that the standard models use the base learning rate for embeddings and normalization gains, whereas Bicameral uses twice that rate. The methods also differ in initialization scale, code/generator learning rates, and weight decay, which raises the issue about effective tuning for competitors.

[1] Zhao J. et al. Galore: Memory-efficient llm training by gradient low-rank projection //arXiv preprint arXiv:2403.03507. – 2024.

[2] Zhu H. et al. Apollo: Sgd-like memory, adamw-level performance //Proceedings of Machine Learning and Systems. – 2025. – Т. 7.

[3] Zmushko P. et al. FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training //arXiv preprint arXiv:2411.07837. – 2024.

[4] Dao T. et al. Monarch: Expressive structured matrices for efficient and accurate training //International Conference on Machine Learning. – PMLR, 2022. – С. 4690-4721.

Questions for authors
  • Does a trained linear generator with the same global layout recover most of the reported quality?
  • How do fine-tuned dense and low-rank baselines perform when embedding/norm learning-rate multipliers and initialization are included in their search?
4 comments

We thank the reviewer for the thorough review of our experimental controls, ablations, and related work. We provide comparisons with GaLore, APOLLO, and Monarch below. We further discuss hyperparameter selection, linear vs. nonlinear generators, and baseline tuning.

Comparison with GaLore, APOLLO

These memory-efficient pretraining methods only compress the optimizer states [6][7][8]. The model's weight memory usage remains uncompressed, whereas Bicameral reduces all of the above by directly reducing trainable parameter count. The two are not mutually exclusive: these methods can be applied to Bicameral to reduce training memory further.

Comparison with Monarch

In Dao et al.[2], Monarch pre-training switches to dense weights before the end of training, and thus does not preserve its memory savings throughout. For a matched-budget comparison, we trained Monarch compressed end-to-end, matching Bicameral's exact stored parameter counts and training state.

All added baselines use warm-up settings from their respective papers and the best point of a LR sweep, trained to 1B tokens.

20 layers, width 3072, 1B tokensStoredTraining statePPL
STD 2B, AdamW1,982M33.3 GB24.00
STD 2B + GaLore (authors' rank d/2)1,982M27.3 GB24.12
STD 2B + APOLLO (authors' rank 256)1,982M19.3 GB36.86
Low rank 2×992M17.4 GB26.11
Monarch 2×992M17.4 GB25.54
Bicameral 2×992M17.4 GB22.34
Low rank 4×496M9.5 GB28.96
Monarch 4×496M9.5 GB25.26
Bicameral 4×496M9.5 GB22.66

Hyperparameter selection

  • STD baselines: hyperparameters selected from existing literature. LR, AdamW hyperparameters and decay schedule follow LLaMA [4][5] and Pythia 1B settings [1]. Initialization uses the fan-in convention with std div. 0.018, similar to small Gaussian initialization at 0.02 found in LLaMA family models.
  • Bicameral's architecture calls for hyperparameters that differ from convention. Qiu et al. [3] showed that structured layers “often require drastically different initialization scales and learning rates”. For Bicameral specifically:
    • Weight decay. We found that AdamW's standard decay fails to regularize Bicameral's generated weights (sec. 5.1): Z grows 16× over 6B tokens, against 1.5× for STD weights. The stronger decay we introduced improves perplexity in all six paired experiments.
    • Initialization. Bicameral updates its materialized weights indirectly. Since latent vectors are short codes, fan-in convention would select initial values that are too large for adamW's weight updates, and cannot balance trainability of the latent vectors vs. the generated parameters. We emprically found smaller inits to be favorable for Bicameral.
    • Tuning. Bicameral's settings were selected from sweeps using small proxy models and were not specifically tuned at the 2B scale.

Q1. Generator linearity

We trained smaller proxy models from scratch with the same global layout. A linear generator recovers most of the quality.

Width 2048, 4× compression, 1B tokensSwishLinear
4 layers30.2830.23
16 layers23.57 (1.2% lower)23.86

Separately, trained 2B models do utilize the nonlinearity. Below we replaced each trained generator with its best linear or quadratic fit, without retraining. We found that the trained models depend on the nonlinearity mainly in attention:

EndpointNonlinearLinear, all matricesLinear, attention onlyLinear, FFN onlyQuadratic Approximation
10B Bicameral 2×14.56143.5127.116.8614.62
10B Bicameral 4×15.7281.281.616.2215.74
40B Bicameral 2× (new)12.49112.4126.420.9212.51

Q2. Baselines optimization

ControlPerplexity
Bicameral 2× with embedding/norm LR at 1× (20 layers, 1B tokens)23.15; STD 2B 24.00; low rank 26.11
Embedding/norm multiplier raised from 2× to 8× (proxy model, 4 layers, widths 512 / 1024)STD −4.0% / −3.6%; Bicameral −4.6% / −3.7%
STD 1B with 2× LR on all parameters (10B tokens)15.29 (1×: 15.91); Bicameral 2× 14.56
STD 2B, ditto14.09
Bicameral at the STD initialization scale (4 layers, 1B tokens)+2.3%
  • Embedding/norm multiplier. The multiplier helps both models equally. With the multiplier at 1×, Bicameral 2× is 3.5% lower in perplexity than STD 2B in this proxy model.
  • LR. Against STD 1B at 2× LR, Bicameral 2× is 4.8% lower in perplexity at 10B tokens. STD 2B also improves to 14.09 at 2x LR. Bicameral 2× remains comparable to STD 2B, and reaches lower loss than STD 1B, with 2x LR baselines.
  • Initialization. For STD, a smaller initialization acts like a higher LR, which the control above covers. Bicameral at the STD initialization scale is 2.3% higher in perplexity.

References

[1] Biderman et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. ICML.

[2] Dao et al. (2022). Monarch: Expressive Structured Matrices for Efficient and Accurate Training. ICML.

[3] Qiu et al. (2024). Compute Better Spent: Replacing Dense Layers with Structured Matrices. ICML.

[4] Touvron et al. (2023a). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.

[5] Touvron et al. (2023b). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.

[6] Zhao et al. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML.

[7] Zhu et al. (2025). APOLLO: SGD-like Memory, AdamW-level Performance. MLSys.

[8] Zmushko et al. (2024). FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training. arXiv:2411.07837.

Thank you again for the time and effort you have devoted to reviewing our work. As the discussion period is drawing to a close, we wanted to kindly follow up to see if our response and additional experiments have addressed your questions and concerns. We would greatly appreciate your feedback and are more than happy to clarify anything that remains open.

I thank authors for a rigorous ablation studies that they provided. I assume, that the vast majority of the questions are covered, however, there are still some concerns about the non-linear generators: they still are not this profitable compared to the linear ones, as the study shows, and the benefits are negligible. However, I appreciate the thorough work done by authors during the discussion period and will raise my score to 3.

Reviewer_hx5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bicameral, a Transformer that stores compact latent codes and generates its weight matrices when needed. This reduces both model storage and optimizer memory during training.

The authors train 2B-effective-parameter models on 10B tokens. Bicameral matches the standard model's perplexity and zero-shot performance with roughly half the stored projection parameters. They also evaluate quantization, memory savings, and optimized inference kernels.

Critical strengths
  1. Bicameral 2X achieves nearly the same perplexity as the standard 2B model (14.56 vs. 14.52) while storing half as many projection parameters. It also clearly outperforms a standard 1B model with similar storage. This is a compelling result.

  2. The paper goes beyond proposing a compressed architecture. The authors measure actual training memory and implement optimized kernels. The reported memory savings and up to 31% faster decoding make the approach more convincing.

Critical weaknesses
  1. The main results come from a single 2B-scale configuration trained on 10B tokens, with one run per configuration. That's still fairly early in language model pretraining. I understand that larger-scale training takes substantial compute and time, and I wouldn't expect a full scaling study. Still, even some preliminary evidence, such as a longer training curve on a smaller model or an analysis of the existing learning trends, would help. I'd like to see some indication that the promising memory–quality trade-off can hold beyond the current setting.

  2. In the actual 10B-token training setup without checkpointing, memory savings are 18 – 24%, rather than the larger savings highlighted under checkpointing. The method also adds compute, and batch-1 prefill is 1.4 – 1.5X slower. These trade-offs are acknowledged, which I appreciate. Still, a clearer memory-versus-throughput comparison would help show when Bicameral is preferable to standard training and inference.

Questions for authors

Refer to Weakness

2 comments

We thank the reviewer for the thoughtful comments and constructive feedback. Below we provide results from larger training runs and a full memory-throughput profile.

W1. Longer training

We trained STD 2B, Bicameral 2× and STD 1B on 40B tokens using a larger batch size, to approximately 20 tokens per effective parameter, the compute-optimal ratio of Hoffmann et al. (2022). Bicameral 2× continues to match STD 2B at this longer horizon.

Table 1: 40B-Token Pre-Training and Zero-Shot Evaluation

ModelSizePPLARC-eARC-cHellaS.PIQAWinoG.LAMB.Zero-Shot Avg
STD 2B1,982M12.6852.928.958.073.156.655.754.2
Bicameral 2×992M12.4956.328.260.074.558.058.455.9
STD 1B973M14.1348.426.551.371.253.149.850.0

W2. Memory vs. throughput

We profiled STD 2B and Bicameral with our optimized kernels and at various batch sizes.

Training. Without checkpointing, Bicameral reduces the weights, gradients and optimizer state and matches the throughput of STD 2B. With checkpointing on, we trade reduced throughput for further reduced memory usage, and can amortize the overhead with larger batch sizes.

ConfigurationMemory saved: Bicameral 2× / 4×Throughput: Bicameral 2× / 4×, relative to STD 2B
No ckpt, micro-batch 4818% / 24%1.00 / 0.99
No ckpt, micro-batch 1623% / 33%1.01 / 1.01
No ckpt, micro-batch 826% / 38%1.04 / 1.04
Per-block ckpt, micro-batch 9630% / 45%0.97 / 0.97
Per-block ckpt, micro-batch 4839% / 56%0.92 / 0.92
Per-block ckpt, micro-batch 1246% / 69%0.75 / 0.73

Inference. We evaluated Bicameral inference on two GPUs with different memory bandwidths and architectures: the NVIDIA H200 (4.8 TB/s) and the NVIDIA RTX PRO 6000 Blackwell (1.8 TB/s). We find that Bicameral inference works well in bandwidth-bound settings by reducing data movement from VRAM to GPU.

Decoding, batchMemory savedThroughput relative to STD 2B, H200Throughput, RTX PRO 6000
146% / 70%1.03 / 1.041.40 / 1.79
445% / 68%1.02 / 1.041.37 / 1.68
844% / 66%1.00 / 1.001.28 / 1.55
1642% / 63%0.87 / 0.901.26 / 1.44
3238% / 56%0.81 / 0.821.15 / 1.25
6431% / 47%0.72 / 0.720.99 / 1.01
PrefillThroughput, H200Throughput, RTX PRO 6000
1 × 8 tokens1.03 / 1.041.44 / 1.82
1 × 32 tokens0.83 / 0.831.28 / 1.51
1 × 64 tokens0.72 / 0.710.92 / 1.01
1 × 512 tokens0.68 / 0.710.82 / 0.88
16 × 512 tokens0.97 / 0.970.98 / 0.98

Decoding and prompts of up to 64 tokens use our optimized kernel that generates and applies the weights fully on-chip; its computational cost grows with the number of tokens, so throughput is highest at 8 tokens. For longer inputs that cannot fit in on-chip cache, the weights are generated and written to VRAM once per forward pass, and that fixed cost becomes a smaller share of the pass as the input grows (throughput 0.97x at batch 16 × 512 tokens).

When Bicameral is preferable.

  • Training without checkpointing: Bicameral saves 18–38% of memory without throuput penalty.
  • Training with checkpointing: when memory is the constraint, Bicameral saves 30–56% at 0.92–0.97 of the throughput, and 46% / 69% at micro-batch 12 at about 75% of the throughput, and enables training of larger models using the same memory budget.
  • Inference: Bicameral always saves memory (31–70%). It is also as fast or faster when the GPU is limited by memory bandwidth. These qualities make Bicameral more viable for resource-limited settings such as local inference, where both memory capacity and bandwidth are at a premium.

A note on relative speed. The decoding speedups in the submission (7% and 31%) were measured on H200s with a 1980 MHz boost clock. The new H200 throughput numbers we present above were run on a different H200 node that has a lower clock frequency of 1560 MHz. We noticed that this caused a discrepancy (STD 2B 1.5% slower, Bicameral 12–13% slower, on average) between the new throughput benchmarks and the original measurements. STD 2B is bound by memory and is less impacted by the slower clock, while Bicameral uses the available compute to save memory, making it more sensitive. While we were unable to test with the 1980 MHz clock within the rebuttal time constraints, we will re-measure all timings on both GPUs when available and update the paper's numbers for the camera-ready version, if accepted.

References

  • Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS.

Thank you again for the time and effort you have devoted to reviewing our work. As the discussion period is drawing to a close, we wanted to kindly follow up to see if our response and additional experiments have addressed your questions and concerns. We would greatly appreciate your feedback and are more than happy to clarify anything that remains open.

Reviewer_nuZmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript proposes Bicameral - a Llama-style architecture that leverages Latent Tensor Rematerialization (LTR), which generates transformer weight matrices from compact latent codes and small learned generators. Each generated matrix is treated as a recomputable activation, so only its codes and generator maintain gradients and optimizer states. Bicameral applies this to all attention and feedforward projections using global tiling, while leaving embeddings and normalization parameters uncompressed. The manuscript studies 2x and 4x compression, code weight decay, and custom kernels. On models with 2B effective parameters trained for 10B tokens, the 2x variant nearly matches the standard baseline in perplexity and zero-shot accuracy while halving stored projection parameters. The manuscript also reports training memory savings and decoding speedups on H200, and shows that the stored codes can be further quantized.

Critical strengths
  1. Reducing memory usage during LLM training is an important practical problem to tackle. The motivation behind the proposed method is clean, as reducing stored parameters also reduces gradient and optimizer-state memory.
  2. The reported experimental results are promising - the 2x model shows little performance degradation while reducing training memory. Also, the paper reports actual memory and latency measurements using custom kernels.
Critical weaknesses
  1. The main 2B models are trained for only 10B tokens, about five tokens per effective parameter. It remains unclear whether the 2x model would still match the standard transformer after substantially longer training.
  2. The paper does not report measurements under FSDP or a 3D (or 5D) parallel training setup. It is unclear how the memory savings and recomputation overhead translate to these distributed training settings.
Questions for authors
  1. Can you report MFU for Bicameral and the standard baseline, clarifying whether generator FLOPs are included?
  2. Do you have an iso-FLOP comparison against the standard transformer, counting the weight generation overhead?
4 comments

We thank the reviewer for the positive assessment of the paper's motivation and experimental results. Below we report new long-horizon training results, parallel training performance, MFU, and Bicameral's overhead in different scenarios.

W1. Longer training

We trained STD 2B, Bicameral 2× and STD 1B on 40B tokens using a larger batch size, to approximately 20 tokens per effective parameter, the compute-optimal ratio of Hoffmann et al. (2022). Bicameral 2× continues to match STD 2B at this longer horizon.

Table 1: 40B-Token Pre-Training and Zero-Shot Evaluation

ModelSizePPLARC-eARC-cHellaS.PIQAWinoG.LAMB.Zero-Shot Avg
STD 2B1,982M12.6852.928.958.073.156.655.754.2
Bicameral 2×992M12.4956.328.260.074.558.058.455.9
STD 1B973M14.1348.426.551.371.253.149.850.0

W2. FSDP and 3D/5D parallelism

We measured Bicameral with TorchTitan (Liang et al., 2025) on 8 H200s, at the paper's 2B shape and at an 8B shape (32 layers, width 4096). The configurations cover FSDP and its combinations with tensor (TP), pipeline (PP) and context parallelism (CP).

  • FSDP all-gathers each block's latent codes and generates the weights locally.
  • TP shards the latent code grids along rows or columns, so each rank generates only its shard of each weight matrix; the small generators are replicated.
  • PP stages hold their own latent codes and generators.
  • CP is unchanged.

Parenthesized numbers are parallelism degrees; peak memory is at micro-batch 1; throughput is measured at the largest micro-batch possible for each model.

Configuration (8 H200s)2B Peak GiB/GPU: STD / Bicameral 2× / 4×2B Throughput: Bicameral 2× / 4× Relative to STD8B Peak GiB/GPU: STD / Bicameral 2× / 4×8B Throughput: Bicameral 2× / 4× Relative to STD
FSDP (8)5.4 / 3.5 / 2.60.94 / 0.9416.6 / 9.3 / 6.30.83 / 0.85
FSDP (4) × TP (2)4.9 / 2.8 / 1.90.94 / 0.9416.6 / 8.9 / 4.90.90 / 0.91
FSDP (2) × TP (2) × PP (2), per-step generation7.0 / 6.4 / 5.50.97 / 0.9722.1 / 19.4 / 16.00.97 / 0.99
FSDP (2) × TP (2) × CP (2), seq 8,1929.6 / 8.4 / 7.60.94 / 0.9524.1 / 19.7 / 17.00.89 / 0.90
  • Memory. Peak memory per GPU is lower in every configuration at both shapes, by up to 44% and 62% under FSDP at 8B scale for Bicameral 2× and 4×, respectively.
  • Recomputation. With weights generated once per optimizer step (PP row), throughput is within 1–3% of STD. The other rows regenerate the weights inside every micro-batch, the setting with the lowest memory. We observed throughput of 0.83–0.92× at 8B and 0.93–0.96× at 2B relative to the STD baselines.

Q1. MFU, and generator FLOPs

Below we report MFU and HFU for our training runs. MFU counts only the FLOPs of the materialized model; HFU adds Bicameral's weight generation. In real training workloads, weight generation incurs a very small overhead.

RunsModelTokens/s/GPUMFUHFUGenerator Share of FLOPs
10BSTD 2B41.2k53.5%53.5%none
Bicameral 2×40.7k52.9%53.1%0.27%
Bicameral 4×40.9k53.2%53.3%0.21%
40BSTD 2B41.8k54.3%54.3%none
Bicameral 2×41.6k54.1%54.2%0.13%

Q2. Iso-FLOP comparison counting generation

Bicameral's generated weights have the same FLOPs as STD 2B, and weight generation adds a small overhead as shown in the table above. Therefore, at equal training tokens and effective parameters, comparisons are approximately iso-FLOP. Below we include FLOP overhead comparisons at different batch sizes. In general, Bicameral's FLOP overhead decreases as batch size increases, which also improves effective throughput.

Training overhead on H200 (values are Bicameral 2× / 4×):

Training Config.Tokens Consumed per MaterializationGeneration Share of Overall FLOPsMeasured Throughput vs. STD 2B
No checkpointing147k0.27% / 0.21%0.99 / 0.99
No checkpointing, larger batch size295k0.13% / 0.11%1.00 / 0.99
Per-block checkpointing, micro-batch 126.1k7.7% / 6.3%0.75 / 0.73
Per-block checkpointing, micro-batch 4824.6k2.0% / 1.7%0.92 / 0.92
Per-block checkpointing, micro-batch 9649.2k1.0% / 0.8%0.97 / 0.97

References

  • Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS.
  • Liang et al. (2025). TorchTitan: One-stop PyTorch Native Solution for Production Ready LLM Pre-training. ICLR.

Thank you again for the time and effort you have devoted to reviewing our work. As the discussion period is drawing to a close, we wanted to kindly follow up to see if our response and additional experiments have addressed your questions and concerns. We would greatly appreciate your feedback and are more than happy to clarify anything that remains open.

Thank you for the experiments. It seems that for fsdp8 there is a non-negligible throughput degradation. For 3d parallel, the degradation seems negligible, but to the best of my knowledge, it is measured on only 1 node, which is not the setting where 3d parallel is often used. I think the manuscript should mention that the memory advantage is achieved at the expense of throughput.

Nevertheless, I believe that these analyses would improve the manuscript. I will keep my position on the Accept side.

Thank you for the recommendation. As a parting thought, since Bicameral was not specifically designed for distributed training, we believe its codebase is currently under-optimized for such setups. However, Bicameral as a memory-reduction technique meaningfully reduces the communication overhead in distributed settings, which should provide throughput benefits. We will tune our kernel for distributed training, and provide an updated analysis in the camera-ready version, if accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.