- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper propses RandomLLM, which tackles the problem of existing method (TicketLLM)'s large randomness, proposes idea of shared randomness to reduce such randomness. Their idea is empirically supported by various experiments across 0.113B to 3B scale, backed up by high-performance execution kernel to realize the benefits of proposed idea during training and inference.
Critical strengths
S1. Clear motivation, simple idea: motivation building on prior work looks very clean, and the idea to tackle such problem (shared randomness) and the execution is simple yet effective, backed up by rigorous experiments.
S2. Efficient execution: Delivering hand-crafted kernels which realizes the efficiency of the proposed idea strengthens the manuscript.
Critical weaknesses
W1. Scalability: Method seems to well-behave under limited training budget, but is not sure if it would scale towards realistic training budget (e.g., 10T+ tokens for B scale models used for production-level LLM pretraining like Qwen)
Questions for authors
Q1. Is there any theoretical guarantee that the concept of shared randomness scales? for instance, would this scale towards 100B parameters or above? I would like to hear from the authors on this topic.
Q2. How would other optimizers (e.g., Muon, Lion, etc.) affect the training dynamics of Ticket(Random)LLMs? Would it be possible to improve training dynamics of STL based approaches on optimization perspective?
8 comments
Response to rebuttal
Thank you for the detailed responses and additional experiments. The new results help clarify the contribution. I have a few thoughts and follow-up questions.
Comparisons with low-bit baselines
I have read Reviewer r6pW's review and agree with the concerns about comparisons with established low-bit methods, and appreciate the added post-training quantization and pruning baselines. These are useful additions that strengthen the empirical evaluation. A comparison with quantization-aware training (QAT) would also be informative, since RandomLLM learns its masks during training, but I recognize that this would require substantial additional work and do not consider it necessary for this rebuttal.
1. Scaling the training budget: held-out evaluation
The extended training trajectories provide useful evidence that RandomLLM continues to optimize through 4B tokens. However, the table reports only training loss. Could the authors also report held-out validation loss and perplexity at the corresponding checkpoints for LLaMA, TicketLLM, and RandomLLM?
This would help assess whether continued optimization translates into improved generalization, and whether the validation gap between the methods narrows, remains stable, or grows as the token budget increases.
2. Scaling depth: trainability with shared randomness
The storage analysis clarifies that, at fixed rank, the random-source budget grows with width and remains independent of depth. My underlying question concerns whether learning remains effective as depth increases while the random source is shared across layers.
The parameter-count calculation and the observation that a rank-one source can yield full-rank masked projections are helpful, but they do not establish optimization behavior in deeper networks. The experiments up to 3B provide initial empirical evidence. Could the authors report the number of transformer layers, hidden width, and feed-forward width for each evaluated scale, so that we can understand how much depth scaling these results cover?
I appreciate the compute constraints; clarifying the depth range already tested would be valuable, along with any observations about training stability as depth increases.
3. Optimizer comparison: tuning and interpretation
Thank you for adding the AdamW, Lion, and Muon comparison. Could the authors clarify whether each optimizer was tuned independently, including its learning rate, weight decay, and relevant optimizer-specific settings, and whether the tuning budgets were comparable?
The results establish that AdamW performs best under the tested configurations. It would also be helpful to hear the authors’ intuition for this advantage: are there properties of mask-score optimization or the thresholding operation that favor AdamW’s updates? Please distinguish any proposed explanation from conclusions directly supported by the experiments.
Response to Reviewer XMuX
0. Comparisons with low-bit baselines
We find the reviewer's suggestion very constructive. We are currently conducting QAT experiments and will report the results as soon as possible.
1. Scaling the training budget: held-out evaluation
Thank you for this helpful suggestion. Unfortunately, we did not save the intermediate 0.7B checkpoints needed to evaluate validation loss and perplexity at different token budgets. We evaluated the saved checkpoints for the 3B models at approximately 0.5B, 1B, 2B, and 3B training tokens. The corresponding held-out validation loss / perplexity results are shown below.
| Approx. tokens | LLaMA | TicketLLM | RandomLLM r=4 |
|---|---|---|---|
| 0.5B | 2.949 / 19.09 | 3.217 / 24.94 | 3.321 / 27.70 |
| 1B | 2.744 / 15.54 | 2.978 / 19.64 | 3.013 / 20.35 |
| 2B | 2.669 / 14.42 | 2.818 / 16.75 | 2.766 / 15.90 |
| 3B | 2.514 / 12.35 | 2.770 / 15.95 | 2.643 / 14.05 |
All three models show a consistent scaling trend: validation loss and perplexity decrease as the training-token budget increases. RandomLLM achieves broadly comparable performance to LLaMA and TicketLLM, and outperforms TicketLLM at approximately 3B tokens, with lower validation loss (2.643 vs. 2.770) and perplexity (14.05 vs. 15.95).
2. Scaling depth: trainability with shared randomness
Thank you for your thoughtful insights and for understanding our computational constraints. The table below lists the four tested model configurations and the validation losses of LLaMA and RandomLLM rank-4.
| Model scale | Transformer layers | Hidden width | Feed-forward width | LLaMA validation loss | RandomLLM rank-4 validation loss |
|---|---|---|---|---|---|
| 0.110B | 12 | 768 | 2,048 | 3.051 | 3.217 |
| 0.335B | 24 | 1,024 | 2,731 | 2.871 | 3.085 |
| 0.7B | 24 | 1,536 | 4,096 | 2.664 | 2.861 |
| 3B | 24 | 3,072 | 8,192 | 2.514 | 2.643 |
Our experiments cover 12 and 24 transformer layers; scaling from 0.335B to 3B increases the hidden and feed-forward widths at a fixed depth of 24 layers. At both tested depths, the recorded rank-4 training losses remained finite and showed an overall downward trend.
These results show that RandomLLM can learn across the tested transformer depths, hidden widths, and feed-forward widths, while sharing one random pool across layers.
3. Optimizer comparison: tuning and interpretation
Given the limited rebuttal time, we explored the following configurations for mask-score optimization. Selected learning rates are shown in bold.
| Optimizer | Candidate learning rates | Fixed optimizer-specific settings |
|---|---|---|
| AdamW | 0.001, 0.003, 0.006 | Betas (0.95, 0.99), epsilon |
| Lion | 0.0001, 0.0003, 0.001 | Betas (0.9, 0.99) |
| Muon | 0.01, 0.03, 0.1 | Momentum 0.95, five Newton–Schulz iterations |
For Muon, matrix-level normalization and shape scaling give a different update scale from AdamW. We tested 0.01, 0.03, and 0.1 to cover one order of magnitude with three roughly logarithmically spaced trials; 0.03 achieved the lowest validation perplexity.
Learning rates were selected independently using validation perplexity. Weight decay and optimizer-specific settings were fixed. All three optimizers applied the same decay to mask scores:
where denotes mask scores, and and are the current and peak learning rates. This normalization gives the same shrinkage at matched schedule positions despite different peak learning rates.
Budgets were identical: three 200-step trials and one fresh 8,000-step run per optimizer, with matched model, data, batch size, schedules, and seeds across optimizers. Other trainable parameters used identical AdamW settings (peak learning rate 0.003).
Mask thresholding depends on score ordering: two scores with gap retain their ordering if . AdamW scales each score's update using its gradient history, , which may help refine connections near cutoffs. Lion uses equal-magnitude nonzero gradient-driven updates within a parameter group, while Muon transforms updates jointly across each score matrix.
Validation perplexity for 0.335B RandomLLM with a shared rank-4 source was recorded every 500 steps:
| Mask optimizer | First recorded step with validation PPL < 25 |
|---|---|
| AdamW | 5,000 |
| Lion | 6,000 |
| Muon | 6,500 |
AdamW reached this threshold with fewer training steps and tokens under the tested configurations.
Response to Reviewer XMuX
Thank you for your careful review and constructive suggestions.
Our source code is available at: anonymous repository.
1. Scaling to realistic pretraining token budgets
We agree that our current training budgets do not establish behavior at production-scale token counts. Our available compute is limited to two H20 GPUs. To examine the trend within this budget, the table summarizes the training trajectories of three 0.7B models from 0.1B to 4B processed tokens (Appendix H, Table 12).
All three models show decreasing training loss as the token budget grows. Specifically, RandomLLM's loss decreases from 6.166 to 2.806, including a further reduction from 2.951 to 2.806 between 3B and 4B tokens. These results show that the shared low-rank random source supports continued learning throughout the measured range.
2. Theoretical guarantees and 100B-scale models
Thank you for raising this question. For a backbone with L blocks, hidden width d, and feed-forward width f, the projection connection count is
P=L(4d2+3df),
while our shared source requires only
Nsource=r(HA+HB).
In our experiments, HA=HB=12,288 across all four model scales. Thus, rank-4 RandomLLM uses the same 98,304 random scalars as the model grows from 0.110B to 3B.
For larger architectures, the pool dimensions can be increased to accommodate larger projections. If the feed-forward and pool dimensions grow proportionally to d, then
PNsource=Θ(Ldr).
At fixed rank, the random-source budget therefore grows linearly with width and remains independent of depth. Mask storage still grows with P.
The illustrative 100B row uses L=80, d=10,240, and f=HA=HB=27,392. It is a parameter-count calculation; the storage column counts only projection masks.
Moreover, at source rank one,
Wj=nj1diag(aj)Mjdiag(bj).
The Gaussian diagonal factors are invertible almost surely, giving rank(Wj)=rank(Mj) (Section 3.3). Thus, a rank-one source does not restrict the masked projection to rank one.
Empirically, the following results from Table 1 show that rank-4 RandomLLM improves from 24.96 to 14.05 PPL and from 35.76% to 39.82% nine-task mean accuracy across the reported runs, while retaining the same random-source budget.
These observations support further investigation at larger scales, although a guarantee of model quality at 100B remains open. Our university laboratory currently lacks the compute to train such models from scratch.
3. Alternative optimizers and training dynamics
Thank you for this suggestion. AdamW achieves the lowest held-out perplexity among the three mask optimizers in this comparison, supporting its use for mask learning in RandomLLM.
Specifically, AdamW reaches 20.66 PPL, compared with 21.56 for Lion and 21.93 for Muon. AdamW also achieves the lowest validation perplexity at every recorded checkpoint, indicating that its advantage in this comparison is observed during training as well as in the final evaluation. These results and the evaluation protocol are included in Appendix F (Table 8) of the revised manuscript.