- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Random Recursive Models (RRM). RRM has a pool of L layers, and at each step, samples one layer from this pool to use, up to T samples. This is like recursive depth / looped language models but with random parameter sharing. The authors provide experiments on a variety of tasks and benchmarks, showing that RRM outperforms comparable baselines.
Critical strengths
- RRM outperforms comparable baselines, demonstrating the approach works well
- The authors provide results on a variety of tasks
Critical weaknesses
The primary weakness is that the evaluation suite is somewhat limited. Although the authors evaluate on a number of datasets with a variety of tasks, the scale is still relatively small. For example, the image classification results are restricted to CIFAR-10, and logic puzzles are used for language modeling (my understanding based on the title of subsection 5.2).
If the authors could expand the experimental section and show comparable results for RRM at greater scale, it would greatly strengthen the work. My suggestions would be to i) expand the image classification results to ImageNet-1k and ii) show results for scaled up language modeling. The language modeling experimental suite can be minimal. I would be very curious how RRM does on language modeling (perplexity plots), especially since their approach does provide the flexibility of more parameters relative to a 1-layer looped tranformer. If the authors could further evaluate the RRM pretrained on language modeling on other benchmarks (like question answering or math word problems), that would be quite interesting too.
Questions for authors
- Why do the authors think this approach works? I find it surprising that randomly sampling layers works so well.
- How similar are the learned weights of each layer?
- Given a trained RRM, is there an optimal ordering of the layers, or does uniformly sampling at inference work the best too?
If the answer is in the paper and I missed it, please point out where.