- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of approximating a controlled memory that evolves over time with linear contractions, under varying regularity assumptions. The algorithm is an online streaming algorithm, and merges ideas from previous works on streaming approximation. The paper provides different guarantees under different setups and regularity assumptions.
Critical strengths
The paper proposes a nice solution in the seemingly hardest scenario of changing control and age dependent contractions, while giving uniform error bounds.
Critical weaknesses
- The paper has poor presentation. The exposition is unreadable, packed with jargon that looks totally LLM generated. Terms are used without any formal or even informal definition. The related work part is completely unreadable (name dropping and almost no explanations) and doesn't clearly situate the work in existing literature.
- It does not explain the relevance or motivation for this problem at all, and does not contain any meaningful discussion of the surrounding literature. It is unclear why this specific setup matters or what previous work (if there is any comparable work) achieved. From what I gather, this setup is totally contrived (see next point). The reference list is very short, which also suggests this problem is not properly motivated.
- Mentioned literature like SHiPPO and HiPPO (zoo) aim to provide a mechanistic interpretation of SSMs, or provide some semantics to the inductive bias that follows from certain memory structures. The main problem is therefore not computational, but expressivity (i.e., successfully matching SSMs). This paper instead aims to find better computational methods, albeit approximate. This seems irrelevant or not yet proven to be interesting for the community.
- I have some reservations about the setup, see the questions section. In particular, I think the regularity assumptions are too strong to be interesting, and the paper doesn't do any work trying to explain why this is relevant.
- The experiments are very limited and do not consider effects of the approximation on the usability of the recurrent memory in learning. Since the paper's claim is computational efficiency (memory wise), it should test if this efficiency comes at the cost of degraded performance. I'm not expecting full fledged SSM results, just some experiment that gives confidence in this direction, even some synthetic tasks.
Questions for authors
- What is the main novelty of the paper? regularity assumptions and the basic techniques are not new. It is unclear if the results follows from new analysis or a clever combination of previous techniques. Please make an effort to highlight the novelty and contributions in a clear manner.
- Table 2 contains previously known constructions, but they hold in different setups and assumptions. The table is not well written and does not provide a clear comparison (e.g., cohort is never defined / explained, what is weak accuracy?). Please provide a better comparison with previous works.
- Misleading result in theorem 2 - assuming is constant implies the first-order modulus might be negligible when is small, making the problem almost trivial (the operator is almost constant in time). Assuming operator difference is lower bounded by some constant makes grow exponentially with , which changes the implications of theorem 2 and it is no longer uniform in . The question is therefore - what does uniformity in give? it seems like a technical claim and it hides the triviality of the claim for small .