- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Building on the theory of MetaPred (which deserves its own separate discussion), the authors present an approach to implement MetaPred with tools that we currently have. The authors present BriLLM which is basically a persistent tensor, coupled with functions to either predict based on the state or update the state, and a controller. Experimental results are done at quite small scale (13M model params and 25M models), but at that scale, BriLLM MetaPred appears to be a stronger language model than GPT. To me, the way to judge this paper is by looking at ablations done within the model and the quality of evaluation / benchmarks. I think the idea is sufficiently novel.
Overall, I observe that the paper makes many assertions about this implementation but does not provide sufficient clear evidence for each assertion. More details on this in the weaknesses section. Consequently, the experiments are not convincing for me; Further, the baseline is likely not the right thing to use - it is parameter matched rather than compute matched.
Critical strengths
1.The paper is quite detailed with math for various components used in the architecture. 2. I like the single step v/s multi step prediction approach to generate consistency. 3. Headline results (11-18% improvement over GPT transformers) is good.
Critical weaknesses
First off, I think the baseline is not convincing - I hope we would compare this approach against the Compute Optimal transformer set up for the amount of compute BriLLM uses.
Second, the paper proposes a fairly complex system - the update function has MANY components, and even after a few reads I don't fully understand the value of each component. It seems like from Table 2, the Modal Nodes are critical but:
- Fast updates + correction off - this seems not useful in practice; Is there an ablation in adaptive mode without fast updates?
- Fixed-point objective removed - this again seems to not matter, however, I don't see any experiments that demonstrate this.
Third, for an engineering of an architecture focused paper, there is no detail on ML infrastructure - I would like to see, at the basic level, training throughput, GPU utilization, Inference statistics etc. Some discussion about scalability/parallelisms that could exist would also be useful.
Questions for authors
Could we start we a significantly simpler approach perhaps? Strip away as many components as possible and study them one by one, then bring them together. Right now, I think there is too much for one paper to study well.