- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper creates a new network architecture that the authors call: Recursive Basis Networks (RBNs). The goal is to decouple functional depth from parameter depth by replacing layer parameters with recursive functional synthesis over a finite shared dictionary.
The authors claim that RBN enables scalable expressivity within a fixed parameter budget.
They provide an approximation error, several ablation studies, a new benchmark they called octagon, scaling behaviours studies.
Critical strengths
Strength 1: The architecture is interesting for parameter-efficient scaling problems, even if the individual components aren't novel, the combination of the three is smart and valuable for many possible applications.
Strength 2: Empirical results are quite exhaustive.
Critical weaknesses
Weakness 1: The expressivity proof in Appendix B1 seems trivial and coming from the triangle inequality and the bound does not look very tight. I think that the convergence rates of are also not clear.
Moreover the constants may depend on depth and it has not been proven they do not. For example if the composed transformation are Lipschitz then we would have a propagation of an error. Resulting in a that is not independant of .
Questions for authors
Overall I found the paper very interesting and will gladly improve my score by 1 or by 2 once the expressivity bound weakness has been resolved.
I expect some more details on the constants and and possibly a tighter bound. The tightness is not an absolute mandatory step for me to increase my score if the answer is otherwise satisfying.
I have a few questions out of curiosity (that do not need answering):
Q1: Do you know what the network actually learn for different recursive step increase? To be more clear I am curious if having more steps will for example make the network learn a finer low frequency structure (or the opposite).
Q2: How is the correlation of the individual basis profiles in the dictionary between each other?
Q3: You gave an upper bound of for the routing configurations, do you empirically know how much of it is actually accessed/used during inference?