- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper study matrix memories for recurrent neural network and propose a non-linear update based on Mobius transformation. They show the proposed method has boundedness guarantees and distinguish from common affine-based updates. Empirical result show that the performance Mobius-based write outperform existing baselines.
Critical strengths
-
Explored non-linear write to memory based on Mobius transformation, which is significantly different from common affine updates. Intuitively, this allow richer interactions between stored associations, and the paper provides a theoretical explanation of its expressivity relative to affine updates.
-
Show theoretical boundedness guarantee and exact chunk-wise algorithm for efficient parallel update.
Critical weaknesses
-
The paper is mathematically dense and poorly written. For example, in Eq. (1), it requires strong prior knowledge on Gated DeltaNet-2 architecture and Eq. (3) - Eq. (6) have a lot of notation not explained clearly. The intuition behind memory update, role of each variable and the headline motivation, the Mobius transformation, is not covered at all.
-
The main result in Table 1 show that the gain is low, and the standard deviation are overlapping, the experiment scope is narrow and limited to Gated DeltaNet-2 only for the main experiment. Table 3 also shows that the runtime for is significantly higher than .
I believe by improving the clarity and explaining the background, this paper can be potentially very good.
Questions for authors
-
Is it possible to repeat the main experiment on another model?
-
Is it possible to have a section to define the task, problem and notation clearly? It is not clear to me the background, for example: what is the structure of Gated DeltaNet-2, and how it link to Section 3 motivation? The paper still have more than 1 page to the page limit (2 for post-rebuttal).
-
The current Section 7 briefly list a huge array of prior works. Given the page limit is 9 pages and there is 1.5 page left, it is better that the author include in the main paper instead of leaving it in the appendix.