acceptodds
Under review as a conference paper at ICLR 2027

Linear Reusable Neural Bases Architecture for Network Compression

Abstract

Memory constraints remain a critical bottleneck in the deployment of large-scale AI models. Parameter sharing across network depth reduces model storage, but repeatedly applying an identical transformation limits flexibility across layers. Inspired by time–memory trade-offs in classical algorithms, we introduce the Linear Reusable Neural Bases (LRNB) architecture, an RNN-based framework that improves parameter efficiency through parameter reuse at the neuron level. Each feedforward residual module is represented as a linear combination of shared neural bases, with depth-specific learnable coefficients and optional shifts providing flexibility across layers. This formulation reduces parameter redundancy across depth and enables the construction of wider and deeper networks within a fixed parameter budget. We further provide a geometric interpretation of the neural bases from a vector-field perspective and extend the framework to linear projection modules. Experiments demonstrate that the LRNB architecture achieves comparable or lower final training loss than independently parameterized baselines while using fewer parameters and maintaining stable training dynamics. These findings support neuron-level reuse as a practical approach to parameter-efficient network design.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Parameter efficiency, Neuron-level reuse, Parameter sharing, Recurrent neural networks, Model compression.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_faLBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Linear Reusable Neural Bases (LRNB), a parameter-sharing framework where neuron weights are represented as linear combinations of a small set of shared basis functions. The authors argue that neuron-level reuse can improve parameter efficiency while maintaining model expressiveness, and evaluate the approach on synthetic regression and WikiText-2 language modeling tasks.

Critical strengths
  • The core idea is simple, intuitive, and easy to implement within existing architectures.
  • LRNB provides substantial parameter reduction while preserving reasonable performance on the presented benchmarks.
  • The paper offers an interesting geometric interpretation of neural networks through reusable bases and vector-field representations.
Critical weaknesses
  • The proposed approach appears closely related to existing cross-layer parameter-sharing methods, particularly Basis Sharing [1], which represents layer weights using shared basis vectors and layer-specific coefficients. The paper does not sufficiently establish the methodological differences or demonstrate a clear advantage over this prior work.
  • The evaluation lacks comparisons with relevant parameter-sharing and model-compression methods, such as [1,2]. Without comparisons against these methods under comparable parameter budgets and evaluation settings, it is difficult to assess the novelty and practical competitiveness of the proposed approach.
  • Key design choices, particularly the gradient rescaling strategy for shared parameters and the parameter-sharing configurations, are not adequately validated. Ablations against alternative optimization strategies are needed to establish the contribution of each component.
  • Although LRNB reduces the parameter count from 34.40M to 19.72M, its reported training time per epoch is approximately 189.5 seconds, compared with 185.4 seconds for the classical Transformer. The paper should provide a more thorough analysis of training and inference latency, memory consumption, and throughput.

Reference

  1. Wang, et al. "Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression." ICLR (2025).
  2. Zhang, Boyang, Daning Cheng, Yunquan Zhang, and Fangming Liu. "Rethinking parameter sharing as graph coloring for structured compression." ICML (2026).
Questions for authors

Check weakness

Reviewer_GrPpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Linear Reusable Neural Bases (LRNB), an architecture that reduces parameters by sharing neurons across the layers of a residual model. Each neuron is treated as a neural basis, with input-side parameters that determine its activation and an output vector that determines its contribution to the residual update. LRNB reuses the same set of neurons across depth while assigning each layer its own learnable coefficient for every neuron. Optional layer-specific shifts provide additional flexibility by changing where individual neurons activate. This design reduces the number of parameters added with each additional layer, allowing the model to be made wider or deeper under a fixed parameter budget. The authors also extend the approach to linear projection layers. Experiments on an FFN-based ResNet for synthetic one-dimensional regression and a small decoder-only Transformer trained on WikiText-2 compare LRNB against standard and parameter-sharing baselines. The authors report that LRNB achieves comparable or lower training and test losses while using fewer parameters in the evaluated settings.

Critical strengths

1. Clear and easy to follow: The neuron-level decomposition and vector-field illustrations make the method intuitive, and the algorithms are simple to reimplement.

2. Lightweight mechanism: Each layer adds only a small coefficient vector (and optional shift) on top of a shared FFN, so the method is easy to integrate into residual architectures.

Critical weaknesses

1. Limited evaluation: The experiments consist of a synthetic 1-D regression task and Transformers of up to 34M parameters trained on WikiText-2 (about 2M tokens). The main text contains no results table; all results are shown as loss curves from a single random seed, which the authors acknowledge in the Reproducibility Statement.

2. Limited novelty: LRNB is a single FFN shared across layers, with a learnable per-neuron scaling vector (and optional shift) for each layer. Pires et al. (2023) already share one wide FFN across layers, and the Remark in Section 4.2 acknowledges that Savarese & Maire (2019) and Wang et al. (2025) also construct layers as learned combinations of shared bases. The paper does not show that moving this idea to the neuron level offers a measurable advantage.

3. Memory usage and inference cost are not reported: The paper is motivated by deployment memory, yet it reports parameter counts and training time but no memory measurements. The authors state that parameter count does not translate directly into memory usage (Section 5) and that inference latency was not evaluated (Appendix C). The 6m variant, which the paper reports as reaching lower training loss than the classical model, takes 495.7 s per epoch versus 185.4 s for the classical model, so the practical efficiency trade-off remains unclear.

4. Confounded comparisons for the expanded variants: The strongest results come from the width-expanded (4m, 6m) and depth-expanded (4L) models, but no shared-FFN baseline of the same width or depth without per-layer coefficients is reported. The gains therefore cannot be attributed to the proposed coefficient mechanism rather than to the added width or depth.

5. Missing ablations: The paper does not ablate the M=1000M = 1000 reparameterization, gradient averaging, or the frozen projection modules from Appendix A, which are used in all experiments. The optional shift, the one added component that is compared, shows no clear improvement over the base model in Figure 3 or Figure 4.

Questions for authors

1. Evaluation scope: Could the authors provide a main results table with final test loss or perplexity for all models, averaged over multiple random seeds? Do the conclusions hold at a larger scale than 34M parameters and WikiText-2?

2. Memory and inference cost: Could the authors report peak memory usage and inference latency for LRNB and the baselines? Since parameter count does not translate directly into memory usage, what practical deployment benefit does LRNB provide?

3. Effect of per-layer coefficients: For the 4m, 6m, and 4L variants, how does LRNB compare with a shared FFN of the same width or depth but without per-layer coefficients?

Reviewer_Qy28OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper finds that a residual feedforward block decomposes into neuron-level neural bases. It proposes Linear Reusable Neural Bases (LRNB), which reuses one shared set of bases and builds each layer as a linear combination with depth-specific coefficients and optional shifts. On a synthetic regression task, LRNB matches or beats a standard ResNet with about one-seventh of the parameters, and on WikiText-2 a wider variant beats a standard decoder-only Transformer in training loss with roughly 12% fewer parameters.

Critical strengths
  1. The paper is clearly written.
Critical weaknesses
  1. The experiments in this paper totally cannot support the conclusions/contributions stated by the author anyway. Firstly, only test the loss here, without any comparison of accuracy. Secondly, even for the loss, in Figure 4, the Test Loss first decreased and then showed a significant increase, reaching as high as 20 in the end, resulting in extremely poor performance. This is clearly overfitting. I do not think this result has any significance.
  2. The graphing is also very poor. Figure 1 could be smoothed/EMA instead of being a stack of various curves.
  3. The experimental scale and ablation are insufficient. Currently, there is only a one-dimensional synthetic function regression task and a 8-layer small language model on WikiText-2. It is completely a toy task. The WikiText-2 dataset is also outdated, and it cannot prove anything.
Questions for authors

Please see weakness. Before being accepted as a formal conference paper, this article needs to undergo significant revisions in its writing, methodology design, and experimental results.

Comments on the paper

1 comment
Area_Chair_mFFVOpenReview ↗

Participate in the discussion period

Dear reviewers,

The authors have replied to your comments and the discussion is close to the end. We encourage you to participate in the discussion to see whether your questions and concerns have been answered by the authors. Thank you!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.