acceptodds
Under review as a conference paper at ICLR 2027

Online Compression of Controlled Memory by Age-graded Rebucketing

Abstract

We construct finite-memory vector approximations of controlled memories in which each past input evolves through age-dependent linear contractions. Future controls arrive online and can induce noncommuting matrix actions, so discard- ing a past input must preserve its response to controls that have not yet been observed. Our algorithms maintain consecutive birth intervals on an increasing age grid and merge adjacent intervals without recovering discarded writes. Under a capped log-age modulus, one vector per interval gives uniform output error ϵ with vector states. With additional second-order age smoothness, positive redistribution between two endpoints improves this sufficient budget to , counting both endpoint vectors. The bounds hold for every bounded signed-vector write stream and common external control path, uniformly over elapsed time and contraction lifetime when regularity constants are fixed. The analysis couples decay-weighted local error with reciprocal-width packing. For the endpoint rep- resentation, a mean-preserving virtual birth coordinate cancels the first variation of the complete chronological matrix product. Small operator-level comparisons use dual bounds on conditional worst-write error to examine the resulting finite- budget approximations without model training.

Primary area:
learning theory
Keywords:
Online memory compression, Streaming algorithms, State-space models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_mAoROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of approximating a controlled memory that evolves over time with linear contractions, under varying regularity assumptions. The algorithm is an online streaming algorithm, and merges ideas from previous works on streaming approximation. The paper provides different guarantees under different setups and regularity assumptions.

Critical strengths

The paper proposes a nice solution in the seemingly hardest scenario of changing control and age dependent contractions, while giving uniform error bounds.

Critical weaknesses
  1. The paper has poor presentation. The exposition is unreadable, packed with jargon that looks totally LLM generated. Terms are used without any formal or even informal definition. The related work part is completely unreadable (name dropping and almost no explanations) and doesn't clearly situate the work in existing literature.
  2. It does not explain the relevance or motivation for this problem at all, and does not contain any meaningful discussion of the surrounding literature. It is unclear why this specific setup matters or what previous work (if there is any comparable work) achieved. From what I gather, this setup is totally contrived (see next point). The reference list is very short, which also suggests this problem is not properly motivated.
  3. Mentioned literature like SHiPPO and HiPPO (zoo) aim to provide a mechanistic interpretation of SSMs, or provide some semantics to the inductive bias that follows from certain memory structures. The main problem is therefore not computational, but expressivity (i.e., successfully matching SSMs). This paper instead aims to find better computational methods, albeit approximate. This seems irrelevant or not yet proven to be interesting for the community.
  4. I have some reservations about the setup, see the questions section. In particular, I think the regularity assumptions are too strong to be interesting, and the paper doesn't do any work trying to explain why this is relevant.
  5. The experiments are very limited and do not consider effects of the approximation on the usability of the recurrent memory in learning. Since the paper's claim is computational efficiency (memory wise), it should test if this efficiency comes at the cost of degraded performance. I'm not expecting full fledged SSM results, just some experiment that gives confidence in this direction, even some synthetic tasks.
Questions for authors
  1. What is the main novelty of the paper? C2C^2 regularity assumptions and the basic techniques are not new. It is unclear if the results follows from new analysis or a clever combination of previous techniques. Please make an effort to highlight the novelty and contributions in a clear manner.
  2. Table 2 contains previously known constructions, but they hold in different setups and assumptions. The table is not well written and does not provide a clear comparison (e.g., cohort is never defined / explained, what is weak accuracy?). Please provide a better comparison with previous works.
  3. Misleading result in theorem 2 - assuming κ\kappa is constant implies the first-order modulus might be negligible when μ\mu is small, making the problem almost trivial (the operator is almost constant in time). Assuming operator difference is lower bounded by some constant makes κ\kappa grow exponentially with μ\mu, which changes the implications of theorem 2 and it is no longer uniform in μ\mu. The question is therefore - what does uniformity in μ\mu give? it seems like a technical claim and it hides the triviality of the claim for small μ\mu.
Reviewer_GD4NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper approximates linear memories whose past writes evolve through age- and control-dependent, possibly noncommuting contractive gates. It uses a merge-only calendar of intervals that widen with age.

This work establishes worst-case budgets for error ϵ\epsilon: O(ϵ−1)O\left(\epsilon^{-1}\right) vectors with one vector per interval under a log-age modulus, and O(ϵ−1/2)O\left(\epsilon^{-1/2}\right) with two endpoint vectors and mean-preserving merges under C2C^2 regularity. Extensions cover readouts, feedback, and a dual formula for worst-write error.

The validation is at operator-level only, on a 2-D noncommuting family (μ=1/64\mu=1/64, T=192T=192).

Critical strengths

An interesting proof mechanism: This paper treats compression as approximating a given operator. The compressor sees only the current gate; it stores no past writes, past controls or propagators. This separates approximation from learning, and the guarantee holds for any continuation of the stream, which heuristic compressed memories in ML don’t offer.

Critical weaknesses
  1. The paper names no model or application whose age-dependent gates must be compressed, and no learned model is involved. Relevance to learned sequence models is asserted, not shown.
  2. The headline lifetime-uniformity is conditional and modest: Assumption 1 scales age variation by μ\mu. If age effects don’t shrink with μ\mu, then Λ∝μ−1\Lambda\propto\mu^{-1} and the budget grows like 1/(μϵ)1/\left(\mu\epsilon\right), which is worse than storing the exact truncated history.
Questions for authors
  1. Which concrete model or application has age-dependent gates that must be compressed, rather than designed to be compressible? Does its age variation scale with μ\mu as Assumption 1 requires, and if not, what budget can you guarantee?
  2. At what lifetime does your width profile beat a geometric calendar, given your constants? If possible, can you add a μ\mu-sweep well beyond 1/641/64 that includes this baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.