- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GliaLLM introduces a glia-inspired framework for jointly learning weight scaling and structured pruning in language models. Two auxiliary networks generate continuous scaling factors and binary pruning masks from backbone weight statistics, and are optimized alternately with the backbone. After training, the scaling factors are folded into the weights and the auxiliary networks are removed. Experiments cover models ranging from 120M to 1.1B parameters and report approximately 23% parameter reduction with modest changes in benchmark performance, alongside measured memory and prefill improvements.
Critical strengths
The framework produces compact models without retaining the auxiliary networks at inference time.
The evaluation includes measured memory consumption, time to first token, prefill throughput, and generation latency. The paper appropriately distinguishes prefill improvements from limited decoding benefits.
Experiments across multiple model sizes, together with multi-seed results and bootstrap analysis, provide useful evidence of stability.
Critical weaknesses
Comparison budgets are insufficiently specified. Table 1 does not report the achieved pruning ratios and complete training or recovery budgets for all baselines. The matched-ratio comparison with Wanda is useful but does not resolve comparability across the other methods. Algorithm 1 includes backbone pretraining before alternating optimization, yet it is unclear whether the reported 110B tokens and GPU hours cover the entire procedure. Table 5 reports 1,067.51 GPU hours for GliaLLM versus 893.93 for the dense model, approximately 19.4% more. These differences require a clearer accounting of the accuracy–compression–compute trade-off.
Accuracy and deployment results are not clearly established for the same final model. The main accuracy results correspond to 23.10% parameter reduction, whereas structural compatibility adjustments and FFN width alignment produce a deployed model with 21.04% reduction. The paper does not report benchmark accuracy for this final exported model or demonstrate numerical equivalence before and after export. Moreover, Table 7 compares compact and full-width versions of the same GliaLLM checkpoint. The reported accuracy preservation and system improvements therefore need to be verified together on the final deployment artifact.
The principal architectural ablations are confounded by pruning ratio. Removing the graph or Mamba module increases the 1.1B model’s pruning ratio from 23.10% to approximately 41.6%–41.9%. The resulting accuracy drops cannot isolate the contributions of these modules from the effect of substantially reduced capacity. Comparisons at matched pruning ratios are needed. A simpler baseline with directly learned gates and scales, or lightweight MLP predictors, would also help establish the value of the two auxiliary networks, which together contain approximately 82M parameters.
Questions for authors
What are the achieved pruning ratios, total training or recovery tokens, and GPU hours for each method? Does GliaLLM’s reported budget include the initial backbone pretraining?
What is the benchmark accuracy of the final exported model with 21.04% parameter reduction, and how closely do its outputs match those before export?
Do the graph and Mamba modules improve performance at matched pruning ratios? How does the full framework compare with simpler gate-and-scale predictors?
How is Overall computed, which checkpoints produce the headline results, and which implementation choices determine the final sparsity?
We thank the reviewer for the constructive follow-up. We have now completed the requested direct per-channel scale-and-gate baseline at the 120M scale. The full GliaLLM framework learned a pruned structure that better preserved predictive accuracy. Compared with directly learning per-channel scales and gates, GliaLLM improved overall
acc_normfrom 0.2796 to 0.2920, a gain of 1.24 percentage points, and scored higher on five of the six benchmarks.The direct approach had an estimated physical pruning rate of 41.60% across the seven projection types, compared with GliaLLM’s reported 23.35%. This more aggressive pruning, together with the lower accuracy, suggests that the direct approach tended to over-prune at the evaluated operating point. These results support the inclusion of the graph network and Mamba in GliaLLM to help preserve the model’s predictive ability during pruning.