acceptodds
Under review as a conference paper at ICLR 2027

RANDOMLLM: HOW LITTLE RANDOMNESS DOES A STRONG LOTTERY TICKET NEED?

Abstract

Strong-ticket training freezes a network's weights at random values and learns only a mask over them. The standard construction for language models samples an independent random weight for every projection, so the random source holds as many independent scalars as the model has connections. We propose RandomLLM, which instead slices every random weight from a single rank- random pool and keeps a separate multivalued mask for every projection. We find that the randomness of the random weight has little influence on the strong ticket that mask learning produces: at source rank , two frozen vectors supply independent scalars, about times fewer than one per connection, and the resulting model reaches perplexity against and against mean zero-shot accuracy at 3B, while a controlled three-seed experiment finds no resolvable effect from either sharing the pool or constraining its rank. Moreover, instead of accelerating this paradigm with dedicated hardware; we give a general-purpose GPU implementation that avoids rebuilding the mask at every forward pass, reducing prefill and decode latency by and and training time by .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Strong Lottery Ticket, Low-bit inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XMuXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propses RandomLLM, which tackles the problem of existing method (TicketLLM)'s large randomness, proposes idea of shared randomness to reduce such randomness. Their idea is empirically supported by various experiments across 0.113B to 3B scale, backed up by high-performance execution kernel to realize the benefits of proposed idea during training and inference.

Critical strengths

S1. Clear motivation, simple idea: motivation building on prior work looks very clean, and the idea to tackle such problem (shared randomness) and the execution is simple yet effective, backed up by rigorous experiments.

S2. Efficient execution: Delivering hand-crafted kernels which realizes the efficiency of the proposed idea strengthens the manuscript.

Critical weaknesses

W1. Scalability: Method seems to well-behave under limited training budget, but is not sure if it would scale towards realistic training budget (e.g., 10T+ tokens for B scale models used for production-level LLM pretraining like Qwen)

Questions for authors

Q1. Is there any theoretical guarantee that the concept of shared randomness scales? for instance, would this scale towards 100B parameters or above? I would like to hear from the authors on this topic.

Q2. How would other optimizers (e.g., Muon, Lion, etc.) affect the training dynamics of Ticket(Random)LLMs? Would it be possible to improve training dynamics of STL based approaches on optimization perspective?

8 comments
AuthorsOpenReview ↗

Response to Reviewer XMuX

Thank you for your careful review and constructive suggestions.

Our source code is available at: anonymous repository.

1. Scaling to realistic pretraining token budgets

We agree that our current training budgets do not establish behavior at production-scale token counts. Our available compute is limited to two H20 GPUs. To examine the trend within this budget, the table summarizes the training trajectories of three 0.7B models from 0.1B to 4B processed tokens (Appendix H, Table 12).

Processed tokensLlama train loss ↓TicketLLM train loss ↓RandomLLM rank-4 train loss ↓RandomLLM − Llama
0.10B6.0926.1926.166+0.073
0.50B3.7134.3014.435+0.722
1.00B3.0593.3703.419+0.360
2.00B2.8423.0723.075+0.233
3.00B2.7362.9652.951+0.214
4.00B2.6132.8192.806+0.192

All three models show decreasing training loss as the token budget grows. Specifically, RandomLLM's loss decreases from 6.166 to 2.806, including a further reduction from 2.951 to 2.806 between 3B and 4B tokens. These results show that the shared low-rank random source supports continued learning throughout the measured range.

2. Theoretical guarantees and 100B-scale models

Thank you for raising this question. For a backbone with LL blocks, hidden width dd, and feed-forward width ff, the projection connection count is

P=L(4d2+3df),P=L(4d^2+3df),

while our shared source requires only

Nsource=r(HA+HB).N_{\mathrm{source}}=r(H_A+H_B).

In our experiments, HA=HB=12,288H_A=H_B=12{,}288 across all four model scales. Thus, rank-4 RandomLLM uses the same 98,304 random scalars as the model grows from 0.110B to 3B.

For larger architectures, the pool dimensions can be increased to accommodate larger projections. If the feed-forward and pool dimensions grow proportionally to dd, then

NsourceP=Θ ⁣(rLd).\frac{N_{\mathrm{source}}}{P}=\Theta\!\left(\frac{r}{Ld}\right).

At fixed rank, the random-source budget therefore grows linearly with width and remains independent of depth. Mask storage still grows with PP.

BackboneProjection connections PPShared rank-4 source scalarsTwo-bit projection masks
Reported 0.335B302.01M98,30475.50 MB
Reported 3B2.718B98,304679.48 MB
Illustrative 100B-scale100.873B219,13625.22 GB

The illustrative 100B row uses L=80L=80, d=10,240d=10{,}240, and f=HA=HB=27,392f=H_A=H_B=27{,}392. It is a parameter-count calculation; the storage column counts only projection masks.

Moreover, at source rank one,

Wj=1njdiag⁡(aj)Mjdiag⁡(bj).W_j=\frac{1}{\sqrt{n_j}}\operatorname{diag}(a_j)M_j\operatorname{diag}(b_j).

The Gaussian diagonal factors are invertible almost surely, giving rank⁡(Wj)=rank⁡(Mj)\operatorname{rank}(W_j)=\operatorname{rank}(M_j) (Section 3.3). Thus, a rank-one source does not restrict the masked projection to rank one.

Empirically, the following results from Table 1 show that rank-4 RandomLLM improves from 24.96 to 14.05 PPL and from 35.76% to 39.82% nine-task mean accuracy across the reported runs, while retaining the same random-source budget.

BackboneLlama PPL ↓Llama Avg. (%) ↑TicketLLM PPL ↓TicketLLM Avg. (%) ↑RandomLLM r=4r=4 PPL ↓RandomLLM r=4r=4 Avg. (%) ↑
0.110B21.1337.6324.5336.4124.9635.76
0.335B17.6639.2321.7738.0921.8736.72
0.7B14.3539.2417.5738.4617.4838.76
3B12.3543.1715.9538.8014.0539.82

These observations support further investigation at larger scales, although a guarantee of model quality at 100B remains open. Our university laboratory currently lacks the compute to train such models from scratch.

3. Alternative optimizers and training dynamics

Thank you for this suggestion. AdamW achieves the lowest held-out perplexity among the three mask optimizers in this comparison, supporting its use for mask learning in RandomLLM.

Mask optimizerHeld-out PPL ↓
AdamW20.66
Lion21.56
Muon21.93

Specifically, AdamW reaches 20.66 PPL, compared with 21.56 for Lion and 21.93 for Muon. AdamW also achieves the lowest validation perplexity at every recorded checkpoint, indicating that its advantage in this comparison is observed during training as well as in the final evaluation. These results and the evaluation protocol are included in Appendix F (Table 8) of the revised manuscript.

Reviewer_XMuXOpenReview ↗

Response to rebuttal

Thank you for the detailed responses and additional experiments. The new results help clarify the contribution. I have a few thoughts and follow-up questions.


Comparisons with low-bit baselines

I have read Reviewer r6pW's review and agree with the concerns about comparisons with established low-bit methods, and appreciate the added post-training quantization and pruning baselines. These are useful additions that strengthen the empirical evaluation. A comparison with quantization-aware training (QAT) would also be informative, since RandomLLM learns its masks during training, but I recognize that this would require substantial additional work and do not consider it necessary for this rebuttal.

1. Scaling the training budget: held-out evaluation

The extended training trajectories provide useful evidence that RandomLLM continues to optimize through 4B tokens. However, the table reports only training loss. Could the authors also report held-out validation loss and perplexity at the corresponding checkpoints for LLaMA, TicketLLM, and RandomLLM?

This would help assess whether continued optimization translates into improved generalization, and whether the validation gap between the methods narrows, remains stable, or grows as the token budget increases.

2. Scaling depth: trainability with shared randomness

The storage analysis clarifies that, at fixed rank, the random-source budget grows with width and remains independent of depth. My underlying question concerns whether learning remains effective as depth increases while the random source is shared across layers.

The parameter-count calculation and the observation that a rank-one source can yield full-rank masked projections are helpful, but they do not establish optimization behavior in deeper networks. The experiments up to 3B provide initial empirical evidence. Could the authors report the number of transformer layers, hidden width, and feed-forward width for each evaluated scale, so that we can understand how much depth scaling these results cover?

I appreciate the compute constraints; clarifying the depth range already tested would be valuable, along with any observations about training stability as depth increases.

3. Optimizer comparison: tuning and interpretation

Thank you for adding the AdamW, Lion, and Muon comparison. Could the authors clarify whether each optimizer was tuned independently, including its learning rate, weight decay, and relevant optimizer-specific settings, and whether the tuning budgets were comparable?

The results establish that AdamW performs best under the tested configurations. It would also be helpful to hear the authors’ intuition for this advantage: are there properties of mask-score optimization or the thresholding operation that favor AdamW’s updates? Please distinguish any proposed explanation from conclusions directly supported by the experiments.

AuthorsOpenReview ↗

Response to Reviewer XMuX

0. Comparisons with low-bit baselines

We find the reviewer's suggestion very constructive. We are currently conducting QAT experiments and will report the results as soon as possible.

1. Scaling the training budget: held-out evaluation

Thank you for this helpful suggestion. Unfortunately, we did not save the intermediate 0.7B checkpoints needed to evaluate validation loss and perplexity at different token budgets. We evaluated the saved checkpoints for the 3B models at approximately 0.5B, 1B, 2B, and 3B training tokens. The corresponding held-out validation loss / perplexity results are shown below.

Approx. tokensLLaMATicketLLMRandomLLM r=4
0.5B2.949 / 19.093.217 / 24.943.321 / 27.70
1B2.744 / 15.542.978 / 19.643.013 / 20.35
2B2.669 / 14.422.818 / 16.752.766 / 15.90
3B2.514 / 12.352.770 / 15.952.643 / 14.05

All three models show a consistent scaling trend: validation loss and perplexity decrease as the training-token budget increases. RandomLLM achieves broadly comparable performance to LLaMA and TicketLLM, and outperforms TicketLLM at approximately 3B tokens, with lower validation loss (2.643 vs. 2.770) and perplexity (14.05 vs. 15.95).

2. Scaling depth: trainability with shared randomness

Thank you for your thoughtful insights and for understanding our computational constraints. The table below lists the four tested model configurations and the validation losses of LLaMA and RandomLLM rank-4.

Model scaleTransformer layersHidden widthFeed-forward widthLLaMA validation lossRandomLLM rank-4 validation loss
0.110B127682,0483.0513.217
0.335B241,0242,7312.8713.085
0.7B241,5364,0962.6642.861
3B243,0728,1922.5142.643

Our experiments cover 12 and 24 transformer layers; scaling from 0.335B to 3B increases the hidden and feed-forward widths at a fixed depth of 24 layers. At both tested depths, the recorded rank-4 training losses remained finite and showed an overall downward trend.

These results show that RandomLLM can learn across the tested transformer depths, hidden widths, and feed-forward widths, while sharing one random pool across layers.

3. Optimizer comparison: tuning and interpretation

Given the limited rebuttal time, we explored the following configurations for mask-score optimization. Selected learning rates are shown in bold.

OptimizerCandidate learning ratesFixed optimizer-specific settings
AdamW0.001, 0.003, 0.006Betas (0.95, 0.99), epsilon 10−810^{-8}
Lion0.0001, 0.0003, 0.001Betas (0.9, 0.99)
Muon0.01, 0.03, 0.1Momentum 0.95, five Newton–Schulz iterations

For Muon, matrix-level normalization and shape scaling give a different update scale from AdamW. We tested 0.01, 0.03, and 0.1 to cover one order of magnitude with three roughly logarithmically spaced trials; 0.03 achieved the lowest validation perplexity.

Learning rates were selected independently using validation perplexity. Weight decay and optimizer-specific settings were fixed. All three optimizers applied the same decay to mask scores:

S←(1−10−5ηtηpeak)S,S\leftarrow\left(1-10^{-5}\frac{\eta_t}{\eta_{\mathrm{peak}}}\right)S,

where SS denotes mask scores, and ηt\eta_t and ηpeak\eta_{\mathrm{peak}} are the current and peak learning rates. This normalization gives the same shrinkage at matched schedule positions despite different peak learning rates.

Budgets were identical: three 200-step trials and one fresh 8,000-step run per optimizer, with matched model, data, batch size, schedules, and seeds across optimizers. Other trainable parameters used identical AdamW settings (peak learning rate 0.003).

Mask thresholding depends on score ordering: two scores with gap dij=∣Si∣−∣Sj∣>0d_{ij}=|S_i|-|S_j|>0 retain their ordering if ∣ΔSi∣+∣ΔSj∣<dij|\Delta S_i|+|\Delta S_j|<d_{ij}. AdamW scales each score's update using its gradient history, m^i/(v^i+ϵ)\hat m_i/(\sqrt{\hat v_i}+\epsilon), which may help refine connections near cutoffs. Lion uses equal-magnitude nonzero gradient-driven updates within a parameter group, while Muon transforms updates jointly across each score matrix.

Validation perplexity for 0.335B RandomLLM with a shared rank-4 source was recorded every 500 steps:

Mask optimizerFirst recorded step with validation PPL < 25
AdamW5,000
Lion6,000
Muon6,500

AdamW reached this threshold with fewer training steps and tokens under the tested configurations.

Reviewer_r6pWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies strong lottery ticket language models. In these models the weights are frozen at random values and never trained, and training learns only a mask that turns each weight off or scales it by 1, 2 or 3. The earlier approach, TicketLLM, gives every layer its own independent random weights. This means billions of random numbers which need to be generated and saved. The proposed RandomLLM instead obtains every layer's random weights out of one shared, low-rank random pool built from two thin random projection matrices A and B. At 3B parameters this is just a few random numbers much smaller than RandomLLM and each layer still learns its own full-size mask. This could be because despite being low rank, applying the mask entry by entry still allows the final weights to be rich.

They test LLaMA-style models from 0.11B to 3B parameters, trained from scratch on FineWebEdu, and compare against their own version of TicketLLM and a normally trained LLaMA. They measure perplexity and accuracy on ten simple zero-shot qa tasks. RandomLLM roughly matches TicketLLM and needs less storage than TicketLLM with its random weights stored explicitly. Neither sharing the pool nor making it low-rank causes a clear loss (the two variants randomllm tries). Most results come from single seed runs, ton smaller scales. Furthermore, I noticed that TicketLLM baseline is a simplified reimplementation, and the paper lacks comparison with standard compression methods such as 2-bit quantization. Many qa scores at these smaller scales are near random guessing, so perplexity is the only reliable signal, and no reasoning, math or code tasks are evaluated. According to me the primary takeaway is that learned mask matters far more than how random the underlying weights are. This takeaway while important is not validated thoroughly.

Critical strengths
  1. The primary finding of the paper is interesting. A shared rank-1 random matrices A and B nearly matches methods that use much larger number of parameters. This is interesting because it challenges the assumption that strong tickets need a large random source, which earlier work handled with special hardware.

  2. Practical payoffs and relation to quantization: Masking entry by entry removes the rank limit (W = Σ diag(a_q) M diag(b_q)). At rank 1 the layer reduces to a 2-bit mask with fixed random row and column scales, which can be computed without forming W. Similar to low bit quantization this leads to a significand drop in memory being used and is quite attractive alternative for low-bit deployment scenarios.

Critical weaknesses
  1. Missing baselines: The paper's motivation is storage and running on ordinary hardware, yet it never compares against methods with the same storage budget: quantization aware training and post training quantization at low bits. I also noticed that TicketLLM baseline is a simplified reimplementation (no Ada-Sup, no binary weights, different initialization). Furthermore, the hypothesis is not verified across larger model scales. Without these comparisons there is no evidence that RandomLLM is useful compared with established compression, so the practical contribution is weak.

  2. Weak supporting evidence for the "randomness isn't necessary" argument: The main table is single-run and confounded by different initializations, training budgets and optimizer restarts, and the best rank was picked after a grid of experiments. The zero-shot tasks are near chance (MMLU is below chance), which is common at lower scales, so accuracy "gains" such as 38.12 vs 37.21 come mostly from noisy BoolQ and COPA scores. This work experiment at meaningful scale. It is known [1] that perplexity is not a good measure of performance at low bits so there should be an analysis on larger scales with qa and reasoning tasks.

[1] Jaiswal, A., Gan, Z., Du, X., Zhang, B., Wang, Z. and Yang, Y., 2024, May. Compressing llms: The truth is rarely pure and never simple. In International Conference on Learning Representations (Vol. 2024, pp. 36973-36993).

Questions for authors
  1. Quantization baselines. How does RandomLLM compare with quantization methods at the same number of bytes? 1-bit methods and 2-bit or 3-bit post-training quantization methods (GPTQ[4], AWQ[5], QuIP#[6], AQLM[7]) would be useful here.
  2. Pruning baselines. RandomLLM is about 50% sparse, so how does it compare with 50% unstructured pruning of the dense LLaMA baseline (magnitude pruning[1], SparseGPT[2], Wanda[3])? Does it still compare well when pruning is combined with quantization so the storage matches?
  3. Reasoning evaluation. Can you evaluate on reasoning-heavy benchmarks such as GSM8K, MATH, HumanEval/MBPP or BBH, ideally with chain-of-thought? Can the method be applied to stronger, reasoning-capable models, for example by learning masks over a pretrained or distilled reasoning model? eg: Qwen3[8] family of reasoning models
  4. Fair comparison to TicketLLM: Can you compare against the published TicketLLM (Ada-Sup optimizer, binary weights)?
  5. Choosing rank and sparsity. How should r be chosen without looking at the grid search results?
  6. Evaluation quality. Many zero-shot scores are near or below chance (e.g., MMLU). Can you report results only for tasks above chance, or add evaluations that separate the methods more clearly?

[1]Han, S., Mao, H. and Dally, W.J., 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.

[2]Frantar, E. and Alistarh, D., 2023, July. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning (pp. 10323-10337). PMLR.

[3]Lan, M., Samy, L., Alshurafa, N., Suh, M.K., Ghasemzadeh, H., Macabasco-O'Connell, A. and Sarrafzadeh, M., 2012, October. Wanda: An end-to-end remote health monitoring and analytics system for heart failure patients. In Proceedings of the conference on Wireless Health (pp. 1-8).

[4] Frantar, E., Ashkboos, S., Hoefler, T. and Alistarh, D., 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

[5] Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.M., Wang, W.C., Xiao, G., Dang, X., Gan, C. and Han, S., 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6, pp.87-100.

[6] Tseng, A., Chee, J., Sun, Q., Kuleshov, V. and De Sa, C., 2024. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. Proceedings of machine learning research, 235, p.48630.

[7] Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A. and Alistarh, D., 2024. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118.

[8] Team, Q., 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 6(7), p.13.

5 comments
AuthorsOpenReview ↗

Respone to Reviewer r6pW

Thank you for your careful review and constructive suggestions.

Our source code is available at: anonymous repository.

1. Quantization baselines at the same storage budget

Thank you for suggesting comparisons at comparable storage budgets. We added 2-, 3-, and 4-bit GPTQ and AWQ baselines. The table compares their saved model sizes and held-out perplexity with RandomLLM.

Model / compression methodSaved model (MB) ↓Held-out PPL ↓
Dense LLaMA, FP321,339.4117.66
GPTQ 4-bit295.1118.15
AWQ 4-bit297.5318.13
GPTQ 3-bit264.8919.24
AWQ 3-bit267.3118.78
GPTQ 2-bit218.0446.21
AWQ 2-bit220.4625.23
RandomLLM, shared rank 4207.7821.87

These results show that RandomLLM provides a competitive storage-quality trade-off near the storage budget of the 2-bit baselines.

Specifically, RandomLLM reaches 21.87 PPL at 207.78 MB, compared with 46.21 PPL at 218.04 MB for 2-bit GPTQ and 25.23 PPL at 220.46 MB for 2-bit AWQ. The 3- and 4-bit baselines achieve lower perplexity than RandomLLM, but require 264.89-297.53 MB, approximately 27-43% more storage.

2. Pruning and pruning-plus-quantization baselines

Thank you for suggesting pruning baselines. We added 50% unstructured magnitude pruning, Wanda, and SparseGPT, together with their AWQ INT2 and INT4 variants.

Storage methodModelProjection sparsitySaved model (MB) ↓Held-out PPL ↓Task mean accuracy ↑
FP32Dense LLaMA0%1,339.4117.6637.60%
FP32Magnitude pruning50%773.1123.1934.84%
FP32Wanda pruning50%773.1620.2936.14%
FP32SparseGPT pruning50%773.1119.1936.31%
AWQ INT4Magnitude + pruning50%257.4523.5734.24%
AWQ INT4Wanda + pruning50%257.4620.5836.38%
AWQ INT4SparseGPT + pruning50%257.4519.4336.06%
AWQ INT2Magnitude + pruning50%219.7041.5432.68%
AWQ INT2Wanda + pruning50%219.7033.6034.10%
AWQ INT2SparseGPT + pruning50%219.7028.7033.97%
2-bit mask; no AWQRandomLLM rank 450%207.7821.8735.34%

These results show that RandomLLM achieves lower perplexity and higher mean accuracy than all three pruning-plus-AWQ INT2 baselines, while using less storage.

Specifically, RandomLLM reaches 21.87 PPL and 35.34% mean accuracy at 207.78 MB. Among the INT2 baselines, SparseGPT achieves the lowest PPL of 28.70, with 33.97% mean accuracy at 219.70 MB. With larger storage budgets, SparseGPT with FP32 weights reaches 19.19 PPL at 773.11 MB, while SparseGPT + AWQ INT4 reaches 19.43 PPL and 36.06% mean accuracy at 257.45 MB.

3. Comparison with the published TicketLLM

Thank you for highlighting the importance of comparison with the published TicketLLM method. We added a TicketLLM baseline using the official 2-bit Ada-Sup masking rule with frozen binary random weights under our training recipe. We also applied the same Ada-Sup masking rule to RandomLLM's shared rank-4 source.

ModelHeld-out PPL ↓
TicketLLM (M-Sup)21.77
RandomLLM (M-Sup)21.87
TicketLLM (Ada-Sup)20.06
RandomLLM (Ada-Sup)20.04

Random-source compression is orthogonal to the choice of masking rule: RandomLLM changes the construction of the frozen random weights while retaining the chosen masking rule. With Ada-Sup, RandomLLM achieves nearly identical observed perplexity to TicketLLM (20.04 vs. 20.06), using only 98,304 random-factor values. Together with the M-Sup results (21.87 vs. 21.77), this supports the effectiveness of our shared low-rank source under both masking rules.

AuthorsOpenReview ↗

Response to Reviewer r6pW

4. Single-run results and experimental confounders

Thank you for raising this concern. Our study includes a controlled comparison across three paired training/random-weight seeds, matching the backbone, mask initialization, random-weight marginals, optimizer, data, and training budget. We reproduce Table 2 below; entries report the mean and sample standard deviation.

ConfigurationLow-rank factorsShared poolPPL ↓Nine-task mean accuracy (%) ↑
TicketLLMNoNo20.784 ± 0.24937.658 ± 0.439
Shared full-rank sourceNoYes21.117 ± 0.32637.284 ± 0.359
Independent rank-4 sourceYesNo21.278 ± 0.32036.980 ± 0.625
RandomLLMYesYes21.192 ± 0.22937.526 ± 0.697

All low-rank configurations use rank 4, and evaluation uses final checkpoints without selecting the best rank. For BoolQ specifically, RandomLLM achieves 60.72 ± 1.31% accuracy versus 59.75 ± 1.50% for the distribution-matched TicketLLM baseline. These repeated runs support preserving comparable downstream performance with a substantially smaller random source. The evidence for this conclusion therefore extends beyond individual gains in Table 1, whose single-run results remain descriptive.

5. Rank and sparsity selection

Thank you for your constructive feedback. Rank selection involves a trade-off between random-source cost and model quality. We favor the smallest rank that meets the required validation quality. The table summarizes the random-source size, construction work, projection storage, and model quality at different ranks.

Random rank rrIndependent random scalarsSource-construction work (relative)Projection storageHeld-out PPL ↓Nine-task mean accuracy ↑
124,5761×75.55 MB22.6336.15%
249,1522×75.60 MB22.0236.73%
498,3044×75.70 MB21.8736.72%
8196,6088×75.90 MB21.3237.48%

Specifically, increasing rank from 1 to 8 reduces PPL from 22.63 to 21.32 and raises mean accuracy from 36.15% to 37.48%, while multiplying the random-source size and theoretical source-construction work by eight. Projection storage increases only from 75.55 to 75.90 MB because the masks dominate it.

For sparsity, we use 50% to match our TicketLLM M-Sup baseline. We also compared sparsity levels of 15%, 35%, 50%, and 75% to examine their effect on model quality.

Projection zero-mask fractionHeld-out PPL ↓Nine-task mean accuracy ↑
15%24.2034.58%
35%22.0136.46%
50%21.2937.41%
75%20.4536.77%

The results support 50% sparsity as a reasonable choice for mean task accuracy and consistency with the baseline. Specifically, 50% achieves the highest observed mean accuracy of 37.41%, while 75% achieves the lowest PPL of 20.45. The preferred sparsity therefore depends on the target metric.

AuthorsOpenReview ↗

Response to Reviewer r6pW

7. Reasoning and mathematics evaluation

Thank you for suggesting these evaluations. We evaluated 0.335B LLaMA and RandomLLM rank-4 models on the full GSM8K and MATH test sets, before and after supervised fine-tuning using their training splits. We used chain-of-thought prompts for both tasks.

Entries report correct answers / total test questions (accuracy).

ModelGSM8KMATH
LLaMA, before fine-tuning0/1,319 (0.00%)0/5,000 (0.00%)
LLaMA, after fine-tuning13/1,319 (0.99%)3/5,000 (0.06%)
RandomLLM rank 4, before fine-tuning4/1,319 (0.30%)0/5,000 (0.00%)
RandomLLM rank 4, after fine-tuning23/1,319 (1.74%)7/5,000 (0.14%)

At the same 0.335B scale, RandomLLM achieves higher observed accuracy after fine-tuning on GSM8K (1.74% vs. 0.99%) and MATH (0.14% vs. 0.06%). The low absolute scores for both LLaMA and RandomLLM indicate that neither model can reliably solve these complex reasoning tasks at the current scale and training budget. Together with the perplexity and controlled three-seed results in Tables 1 and 2, these findings support our central claim that substantially reducing the randomness budget preserves comparable model quality within the evaluated range.

6 & 8. Evaluation quality and larger models

Thank you for these suggestions. We agree that MMLU provides limited discrimination among the evaluated models under our current scales and training budgets. The LLaMA baselines trained in this study also score only 22.84–23.62% (Appendix G, Table 9). We have moved MMLU results to Appendix G and excluded this task from the main aggregate, recomputing the nine-task averages consistently for all models while retaining the complete results for transparency.

Other tasks show improvement as model size increases. With rank fixed at 4, RandomLLM’s ARC-E accuracy increases from 34.21% to 42.11%, and HellaSwag accuracy from 26.35% to 31.00%, as the model grows from 0.110B to 3B (Table 1). Together with the controlled three-seed results reproduced in response 4, these observations support our central claim that mask learning remains effective with a substantially smaller shared random source within the evaluated range. The 0.7B rank-4 model also shows continued learning: training loss decreases from 2.951 to 2.806 between 3B and 4B processed tokens (Appendix H, Table 12).

Our laboratory currently lacks the resources to train larger models from scratch. For context, published TicketLLM reports experiments up to 1.3B, whereas our experiments extend to 3B. TicketLLM, Section 4.4 We will explore larger models and knowledge distillation from stronger teachers to guide mask learning over our frozen shared random source, with further evaluation of reasoning, mathematics, and code-generation performance.

Reviewer_DE89OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents two contributions, both building on the TicketLLM paper:

  1. A random-weight initialization method that avoids the need for specific hardware while keeping model weight storage dominated by the size of the low-bit mask.
  2. GPU inference speedups to mitigate the extra costs of the TicketLLM and RandomLLM-style training, largely independently of the initialization method.

They show rigorous ablations across model size and random weight hyperparameter (rank) settings showing low sensitivity to the random weight hyperparameters across the model size spectrum and that a low rank is sufficient to recover the accuracy of TicketLLM. They additionally show benchmarks for improvements in speed compared to the PyTorch baseline (similar to the TicketLLM baseline with the additional cost of materializing the random-weight matrix).

The paper was overall clearly written, and presented a genuinely novel approach that builds on TicketLLM. Both of the core contributions: a shared-pool, low-rank, random weight initialization and PyTorch training/inference speedup for the model are well thought out, described and tested. My only concern comes in the combination of the two: it is unclear where the improvement over TicketLLM materializes (they have specifically stated they are not comparing to other low-bit methods). Their mask-packing and weight caching clearly speed up TicketLLM inference, however, with the fact that they still materialize the full weight initialization at the beginning of training/inference, the low-rank weight factorization is orthogonal. If the claim is in parameter storage, it is unclear that the weight factorization is the optimal choice. If we are only initializing weights once, seeding a GPU random-number generator (negligible storage) and/or using e.g. a circulant matrix initialization (significantly less storage) of the shared pool may be acceptable, and no comparison in presented to alternatives.

Critical strengths

The paper is clearly written, without excess technical detail, quickly revealing the intuition behind the approaches.

The intuition that fully random weight initialization for lottery-ticket models is very interesting and the implementation speedups are impressive.

Critical weaknesses

The paper gives the initial impression that the two contributions are not independent, but it is unclear how they build upon each other.

I have unresolved questions about the choice of random weight initialization. The finding that a full-rank random weight matrix is not necessary is in itself interesting, but doesn't seem sufficient to justify the specific choice of their method, especially lacking comparison to other random weight initialization methods.

Questions for authors

What constraints make the choice of low-rank random weight initialization the best choice?

1 comment
AuthorsOpenReview ↗

Response to Reviewer DE89

Thank you for your careful review and constructive suggestions.

Our source code is available at: anonymous repository.

1. Relationship between the two contributions

Thank you for pointing this out. Random-source compression and our GPU execution optimizations are orthogonal.

At the software level, we investigate how much independent randomness strong-ticket learning requires. RandomLLM reduces the random-source budget through a shared low-rank random pool while retaining separate masks for each projection (Figure 1). Table 1 reports the resulting storage and model quality, while Table 2 separately examines the effects of sharing the pool and reducing its rank.

At the hardware level, we optimize execution on general-purpose GPUs. Mask packing stores two-bit masks and unpacks them within the GPU kernel, avoiding repeated mask computation. Weight caching stores the weights after masking and reuses them during inference (Section 3.5). Figure 2 shows the speedups over the PyTorch baseline within RandomLLM, with random weights materialized at initialization. These optimizations are also applicable in principle to TicketLLM. We have clarified this distinction in the introduction and Section 3.5.

2. Storage advantage and alternative random sources

Thank you for this insightful suggestion. At the software level, our central question is how much independent randomness is needed to construct an SLT model. Our method controls this budget through low-rank factorization and sharing. Following your suggestion, we added Gaussian and marginal-matched circulant sources as alternative constructions. As shown in the table, both achieve lower held-out perplexity with fewer stored source values, demonstrating that circulant sources are also effective alternatives. We have added the circulant constructions and their quality and GPU construction-time comparisons to the appendix.

Random-source constructionHeld-out PPL ↓PyTorch construction time (ms) ↓Speedup vs. Gaussian circulant
Shared rank-4 factors21.860.703≈7.6×\approx 7.6\times
Shared Gaussian circulant21.665.3091.0×1.0\times
Shared marginal-matched circulant20.975.311≈1.0×\approx 1.0\times

At the hardware level, circulant construction uses cyclic shifts, implemented here through indexing operations, while our low-rank construction uses matrix multiplication. The latter benefits from optimized GPU matrix-multiplication routines and constructs the pool approximately 7.6×7.6\times faster in the PyTorch comparison shown above. This result highlights the execution cost of the indexing-based implementation. Specialized hardware, such as ASIC designs for circulant generation, could provide another way to execute these alternative constructions efficiently.

3. Justification for choosing low-rank initialization

Thank you for raising this question. First, we chose low-rank initialization because rank directly controls the number of independent random values, r(HA+HB)r(H_A+H_B), while keeping the architecture and mask parameterization unchanged (Section 3.1 and Figure 1). This provides a direct way to investigate how much randomness an SLT model needs. Table 1 and the additional circulant results in our response to point 2 show that effective SLT models can be learned without independently sampling a random weight for every connection.

Second, the low-rank source is constructed through matrix multiplication, which benefits from optimized GPU routines. As shown in our response to point 2, this construction is approximately 7.6×7.6\times faster than circulant indexing in the measured PyTorch implementations. These properties make low-rank initialization a controllable and practical choice. The appendix comparison also demonstrates effective alternative random-source constructions.

Comments on the paper

1 comment
Area_Chair_FRzSOpenReview ↗

AC message

Dear Reviewers

As we are now more than halfway through the discussion period, I encourage all reviewers to engage in the responses and discuss should there be more questions.

---AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.