acceptodds
Under review as a conference paper at ICLR 2027

GliaLLM: Beyond Neuronal Spotlight —— Glia-Inspired Structural Learning in Large Language Models

Abstract

Large language models (LLMs) typically rely on parameter scaling while maintaining fixed architectures, which leads to considerable redundancy in both model parameters and computation. Inspired by astrocyte-mediated bidirectional synaptic plasticity and microglia-mediated selective synaptic pruning, we propose the artificial Neuron–Astrocyte–Microglia (NAM), a new computational model for neuron-level structure learning in neural networks. Building on NAM, we develop the GliaLLM framework, in which Astrocyte Net (AS-Net) and Microglia Net (MG-Net) jointly learn layer- and projection-specific continuous scaling factors and binary gates from the LLM’s linear-layer weights during training. These modulation signals are then computed once offline and folded into the LLM to produce a deployable pruned subnetwork. We introduce a three-stage alternating optimization strategy to jointly learn model parameters and network structure. Relative to the pretrained Llama-style-1.1B baseline, GliaLLM achieves a 23.1% parameter reduction ratio (PRR), while incurring only a 0.006 decrease in aggregate across six benchmarks. These results demonstrate that glia-inspired neuron-level structure learning can yield deployable LLM subnetworks that lower peak GPU memory usage across workloads and reduce TTFT for long-input inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Glia-inspired Structural Learning, Large Language Models, Structured Pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4Xs3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GliaLLM introduces a glia-inspired framework for jointly learning weight scaling and structured pruning in language models. Two auxiliary networks generate continuous scaling factors and binary pruning masks from backbone weight statistics, and are optimized alternately with the backbone. After training, the scaling factors are folded into the weights and the auxiliary networks are removed. Experiments cover models ranging from 120M to 1.1B parameters and report approximately 23% parameter reduction with modest changes in benchmark performance, alongside measured memory and prefill improvements.

Critical strengths

The framework produces compact models without retaining the auxiliary networks at inference time.

The evaluation includes measured memory consumption, time to first token, prefill throughput, and generation latency. The paper appropriately distinguishes prefill improvements from limited decoding benefits.

Experiments across multiple model sizes, together with multi-seed results and bootstrap analysis, provide useful evidence of stability.

Critical weaknesses

Comparison budgets are insufficiently specified. Table 1 does not report the achieved pruning ratios and complete training or recovery budgets for all baselines. The matched-ratio comparison with Wanda is useful but does not resolve comparability across the other methods. Algorithm 1 includes backbone pretraining before alternating optimization, yet it is unclear whether the reported 110B tokens and GPU hours cover the entire procedure. Table 5 reports 1,067.51 GPU hours for GliaLLM versus 893.93 for the dense model, approximately 19.4% more. These differences require a clearer accounting of the accuracy–compression–compute trade-off.

Accuracy and deployment results are not clearly established for the same final model. The main accuracy results correspond to 23.10% parameter reduction, whereas structural compatibility adjustments and FFN width alignment produce a deployed model with 21.04% reduction. The paper does not report benchmark accuracy for this final exported model or demonstrate numerical equivalence before and after export. Moreover, Table 7 compares compact and full-width versions of the same GliaLLM checkpoint. The reported accuracy preservation and system improvements therefore need to be verified together on the final deployment artifact.

The principal architectural ablations are confounded by pruning ratio. Removing the graph or Mamba module increases the 1.1B model’s pruning ratio from 23.10% to approximately 41.6%–41.9%. The resulting accuracy drops cannot isolate the contributions of these modules from the effect of substantially reduced capacity. Comparisons at matched pruning ratios are needed. A simpler baseline with directly learned gates and scales, or lightweight MLP predictors, would also help establish the value of the two auxiliary networks, which together contain approximately 82M parameters.

Questions for authors

What are the achieved pruning ratios, total training or recovery tokens, and GPU hours for each method? Does GliaLLM’s reported budget include the initial backbone pretraining?

What is the benchmark accuracy of the final exported model with 21.04% parameter reduction, and how closely do its outputs match those before export?

Do the graph and Mamba modules improve performance at matched pruning ratios? How does the full framework compare with simpler gate-and-scale predictors?

How is Overall computed, which checkpoints produce the headline results, and which implementation choices determine the final sparsity?

Reviewer_SRNdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the artificial Neuron–Astrocyte–Microglia (NAM) model, inspired by astrocyte-mediated synaptic modulation and microglia-mediated pruning, for neuron-level structural learning in LLMs. Building on NAM, GliaLLM uses AS-Net and MG-Net to generate continuous scaling factors and binary retention gates, which are learned jointly with model parameters through a three-stage alternating optimization process. After training, these signals are folded into the backbone to produce a deployable compact subnetwork.

Critical strengths

This paper combines continuous modulation with selective structural pruning, avoiding premature hard pruning decisions. Its offline signal integration and physical pruning yield a static, hardware-compatible subnetwork without auxiliary modules at inference time.

Critical weaknesses
  1. The innovation in the method of this paper is a combinational one: graph coupling + Mamba timing + STE binary gate + exp/tanh scaling + three-stage alternation. It can be implemented in engineering and a compact model can be derived, but the proportion of biological narrative is relatively high, so the innovation of this method in this paper is a major issue.
  2. The construction of the graph structure in IG-GSC is unclear.
  3. The calculation of the Overall metric reported in Table 1 is unclear.
  4. The ablation study lacks an AS-Net-only experiment.
  5. Table 1 does not report the pruning ratios of the baseline methods.
  6. The appendix shows an approximately 19.4% increase in computational cost, while the main text emphasizes comparable training time. 7.The results of the main experiments are inconsistent with those of the multi-seed experiments.
Questions for authors

See Weakness.

Reviewer_RHbLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GliaLLM, a framework for jointly learning language-model weights and channel-level structure. Two auxiliary networks generate continuous scaling factors and binary gates from row-norm statistics of the backbone weights, using MLP encoders, graph attention, and Mamba-2 blocks. Training alternates between updating the scaling network, the gating network, and the backbone. The final modulation signals are folded into the weights, and the auxiliary networks are discarded for deployment. On a custom Llama-style 1.1B model, the paper reports 23.10% parameter reduction with Overall acc_norm decreasing from 0.358 to 0.352. A hardware-aligned export achieves 21.04% reduction and measured memory and prefill benefits. Additional experiments examine smaller models, component ablations, and training-seed variability.

Critical strengths
  1. The framework addresses deployable structured compression rather than weight sparsity alone. The auxiliary networks are removed at inference, and the exported model uses standard dense operators. Measurements on an RTX 3090 demonstrate lower peak memory and faster prefill, while the paper appropriately limits its claims about end-to-end acceleration.

  2. The empirical evaluation includes multiple model settings, training-seed results, and paired-bootstrap analysis. The 120M AS-Net ablation is particularly informative: removing continuous scaling worsens accuracy while also reducing compression. These results provide useful evidence that coordinating scaling and selection can benefit the learned subnetwork.

Critical weaknesses
  1. The experiments do not sufficiently establish the added value of the proposed auxiliary architecture.

Removing IG-GSC or IG-TSE increases PRR from 23.10% to approximately 42%, confounding the component comparison with a substantial change in compression. These results show that the modules influence the learned operating point, but do not establish a better quality–compression trade-off at matched model size. A simpler baseline with directly learned channel scales and gates would further clarify the need for graph attention and Mamba. For the external methods in Table 1, achieved parameter counts and recovery-training budgets are also insufficiently documented. Together, these gaps limit the evidence that the proposed complexity improves upon simpler or existing approaches under comparable conditions.

  1. Important details linking the headline accuracy result to training cost and the deployed model remain unclear.

The main result pairs 0.352 Overall acc_norm with 23.10% pre-alignment PRR, whereas the hardware measurements use an export with 21.04% PRR. It is not explicit whether the reported accuracy applies to that final export, and no equivalence check is provided for the complete conversion. The allocation of the stated 110B-token budget between initialization and alternating optimization also needs clarification. Finally, the aggregation underlying Overall acc_norm and the relationship between the main-table checkpoint and the runs in Appendix Table 6 are not sufficiently specified. These are consequential reporting gaps because they affect the claimed quality–compression–cost trade-off.

Questions for authors
  1. Please specify the achieved parameter count and calibration or recovery-training budget for each method in Table 1. Does GliaLLM’s stated 110B-token budget include both the initialization in Eq. 8 and all subsequent alternating stages, and how are tokens allocated between them?

  2. Do IG-GSC and IG-TSE improve quality at approximately matched PRRs? Comparisons at matched compression levels, including a baseline with directly learned channel scales and gates, would help establish the contribution of the auxiliary architecture.

  3. Does the 0.352 Overall score apply to the final hardware-aligned export with 21.04% PRR? Please provide either post-export benchmark results or a numerical equivalence check showing that the complete conversion preserves the evaluated model’s outputs.

  4. How is Overall acc_norm computed across benchmarks and subtasks? Please also identify the run and checkpoint used for the main 0.352 result and explain how it relates to the three 1.1B runs in Appendix Table 6.

3 comments

We thank the reviewer for the constructive follow-up. We have now completed the requested direct per-channel scale-and-gate baseline at the 120M scale. The full GliaLLM framework learned a pruned structure that better preserved predictive accuracy. Compared with directly learning per-channel scales and gates, GliaLLM improved overall acc_norm from 0.2796 to 0.2920, a gain of 1.24 percentage points, and scored higher on five of the six benchmarks.

120M methodOverallARCHellaSwagPIQAOpenBookQAWinoGrandeMMLU
Direct scales and gates0.27960.3410.3060.5990.2560.5090.2678
GliaLLM0.29200.3750.3460.6120.2760.5050.2780

The direct approach had an estimated physical pruning rate of 41.60% across the seven projection types, compared with GliaLLM’s reported 23.35%. This more aggressive pruning, together with the lower accuracy, suggests that the direct approach tended to over-prune at the evaluated operating point. These results support the inclusion of the graph network and Mamba in GliaLLM to help preserve the model’s predictive ability during pruning.

Thank you for clarifying the training budgets, score aggregation, and checkpoint used for the main result. The direct evaluation of the hardware-aligned export addresses my deployment question. The exported model achieves an overall acc_norm of 0.352 with 21.04% fewer parameters.

I also appreciate the new baseline with directly learned per-channel scales and gates. At the 120M scale, this baseline scores 0.2796 at an estimated 41.60% pruning rate, compared with 0.2920 at 23.35% for GliaLLM. Together with the 1.1B ablations, this shows that the auxiliary architecture influences the amount of pruning under the reported training setup. The difference in retained model size leaves open my main question about whether graph attention and Mamba improve accuracy at a comparable deployed size and training budget. My recommendation is 2 (Weak Reject).

We thank the reviewer for highlighting the role of retained model capacity in interpreting the ablation results. We revisited the 1.1B checkpoints and found that, at similar structural pruning rates, the full GliaLLM surpassed the variant without graph attention and came close to the accuracy of the variant without Mamba much earlier in training.

In the closest match by structural PRR (see Figure 7 in the appendix for the gate retention curves for each projection type), the full GliaLLM reached 41.56% PRR and an Overall acc_norm of 0.3187 at cycle 9,300. After only 24.5% as many training cycles as the ablations evaluated at cycle 38,000, it outperformed the variant without IG-GSC (41.59% PRR, 0.2747) by 4.39 percentage points and was within 0.30 percentage points of the variant without IG-TSE (41.93% PRR, 0.3217). By cycle 12,500, after 32.9% as many training cycles, the full model reached 0.3257 at 41.01% PRR, surpassing both ablations.

These results provide further empirical support for the joint use of graph attention and Mamba in GliaLLM. At comparable retained capacity as measured by structural PRR, the full model achieved higher accuracy than both ablations after roughly one-third as many training cycles. We will include these comparisons in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.