acceptodds
Under review as a conference paper at ICLR 2027

Infinite From Finite: Recursive Basis Networks

Abstract

Depth is a fundamental axis of neural scaling, yet deep neural networks typically obtain deeper computation by stacking more parameterized layers, tightly coupling functional depth to parameter depth. In this paper, we explore a new route to neural scaling, where functional depth grows through synthesis from a finite parameter reservoir. We propose Recursive Basis Networks (RBNs), which achieve dynamic functional synthesis by recursively recomposing a shared basis dictionary through input-conditioned states. This enables RBNs to synthesize heterogeneous functional transformations without layer-specific parameters, turning depth into a runtime-controllable dimension while keeping the transformation parameter budget fixed. Therefore, RBNs transform parameter accumulation into functional composition, yielding a large compositional space from finite primitives. Across diverse approximation and representation-learning settings, RBNs demonstrate the effectiveness of dynamic functional synthesis, with strong expressivity, parameter efficiency, and flexible depth scaling. These results establish it as a general principle for deriving expandable functional complexity from finite parameterization. Our code will be released on our website.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Dynamic Functional Synthesis, Compositional Modeling, Neural Architectures, Parameter-Efficient Learning, Recursive Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cN7yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper creates a new network architecture that the authors call: Recursive Basis Networks (RBNs). The goal is to decouple functional depth from parameter depth by replacing layer parameters with recursive functional synthesis over a finite shared dictionary.

The authors claim that RBN enables scalable expressivity within a fixed parameter budget.

They provide an approximation error, several ablation studies, a new benchmark they called octagon, scaling behaviours studies.

Critical strengths

Strength 1: The architecture is interesting for parameter-efficient scaling problems, even if the individual components aren't novel, the combination of the three is smart and valuable for many possible applications.

Strength 2: Empirical results are quite exhaustive.

Critical weaknesses

Weakness 1: The expressivity proof in Appendix B1 seems trivial and coming from the triangle inequality and the bound does not look very tight. I think that the convergence rates of εbasis,com\varepsilon_{basis,com} are also not clear.

Moreover the constants C1,2C_{1,2} may depend on depth LL and it has not been proven they do not. For example if the composed transformation are Lipschitz then we would have a propagation of an error. Resulting in a CC that is not independant of LL.

Questions for authors

Overall I found the paper very interesting and will gladly improve my score by 1 or by 2 once the expressivity bound weakness has been resolved.

I expect some more details on the constants C1C_1 and C2C_2 and possibly a tighter bound. The tightness is not an absolute mandatory step for me to increase my score if the answer is otherwise satisfying.

I have a few questions out of curiosity (that do not need answering):

Q1: Do you know what the network actually learn for different recursive step increase? To be more clear I am curious if having more steps will for example make the network learn a finer low frequency structure (or the opposite).

Q2: How is the correlation of the individual basis profiles in the dictionary between each other?

Q3: You gave an upper bound of KTK^T for the routing configurations, do you empirically know how much of it is actually accessed/used during inference?

Reviewer_LuFJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Recursive Basis Networks (RBNs), which combine KAN-style basis functions with input-conditioned coefficient composition and recursive parameter sharing. RBN can be viewed as a looped KAN with finer-grained, basis-wise routing similar to MoE. The architecture allows functional depth to increase without introducing additional step-specific transformation parameters.

The authors evaluate RBN on synthetic function approximation, PDEBench, and visual representation learning. The results show promising approximation accuracy, parameter efficiency, and flexible test-time depth scaling.

Critical strengths
  1. Interesting architecture. Combining basis-wise dynamic composition with recursive parameter sharing is an interesting idea with sufficient architectural novelty. It allows the same parameterized cell to produce different transformations across recursive steps.

  2. Promising empirical results. RBN shows strong parameter efficiency on the Octagon benchmark and competitive results across additional function approximation tasks. The ViR experiments also suggest that the approach can extend to visual representation learning, with performance improving as inference depth increases.

Critical weaknesses

My main concern is the necessity of the global dictionary WW. Why not simply use the Dynamic Composer to directly predict the coefficients over the TT B-spline basis functions, without introducing the intermediate dictionary of size KK? This would still allow input-conditioned basis composition and recursive parameter sharing. It is unclear what additional benefit the dictionary WW provides, either theoretically or empirically. I can roughly see why we need input-dependent SS, based on the K=1K=1 RBN results, and why we need basis functions, based on the comparisons with MLP/RNN. However, the necessity of WW remains unclear.

Second, I noticed that the MoE baseline does not use the same B-spline basis functions, but instead uses much simpler linear experts. In fact, if we used B-spline bases for MoE, we could have a model where pk(h)p_k(h) is shared across all TT basis functions, and the router selects entire spline-based experts. This would effectively be a restricted case of RBN and could serve as a useful control for evaluating the granularity of routing.

Finally, instead of only comparing against a tanh RNN, since RBN also has an attention-like component (attention-style coefficient gating, where the query is the hidden state, the keys are π\pi, and the values are WW), it would be more informative to include a Transformer-like baseline. The authors report UT-ACT in the ACT setup, but why is it not included in the fixed-depth experiments in Fig. 4?

The ViR vs. ViT comparison is related, but it is not parameter-matched. In Table 1, ViR at L=8L=8 achieves 54.82% average accuracy versus 51.48% for ViT-base, but uses 130.95M parameters compared with 85.88M. Appendix B.2 constructs a parameter-matched ViR (86.32M) for the wall-clock comparison, but does not report its linear-probing accuracy.

On the dataset side, the experiments are largely synthetic, which is related to a more broader concern about the need to choose appropriate basis functions for each domain. For example, how would this approach work in discrete domains? Octagon and PDEBench involve functions that are naturally suited to B-spline approximation. It is unclear how we would choose suitable basis functions, or even the hyperparameters KK and TT, for other domains. The ViR experiment is particularly important in this regard, but the lack of parameter matching weakens its evidence.

Questions for authors
  1. How is leml_{\mathrm{em}} in Eq. 6 calculated? Is it directly learned? If so, how does it generalize to unseen step indices in the test-time depth-scaling experiment? From Fig. 2, I can only guess that it is something like

    lem=l⋅em.l_{\mathrm{em}}=l\cdot e_m.

  2. There seems to be an inconsistency in Table 9. Fixed-depth RBN at L=4L=4 uses 0.2744M FLOPs, while RBN-ACT with a maximum depth of 4 uses 0.3979M FLOPs. How does ACT, which can potentially execute fewer than four steps, incur substantially more FLOPs than the fixed-depth model?

Reviewer_aUYgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Recursive Basis Networks architecture that allows for increasing functional depth while keeping fixed parameter count. The approach is conceptually similar to Kolmogorov-Arnold Networks as well as dynamic routing in MoE. The experimental evaluation involves a toy example, as well as application to vision.

Critical strengths

The paper provides an interesting extension of ideas underpinning recursive computation, KANs, and MoEs.

Critical weaknesses

The critical weakness of the manuscript lies in comparison of the claimed contribution vs existing methods, both at the conceptual level, as well as experimentally. While the authors mention some recursive method like Universal Transformers, more recent results in this direction (e.g. Hierarchical Recursive Models) are missing. More importantly, dynamic or adaptive extensions of KANs are not discussed conceptually or evaluated empirically (e.g. InfinityKAN, Residual-Gated Adaptive KAN). Separately, the experimental comparison only uses standard ViT, and is missing e.g. Vision-KAN and other KAN-based vision models, not allowing for disentangling how much of the difference comes from the broad type of architecture, and how much from specific contribution of this manuscript.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.