- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether keeping a model's current predictions is enough for future learning and shows that two models can make the same predictions but react differently to new data. The authors explore this question through theoretical analyses of several models, mainly supported by mathematical proofs and some numerical checks.
Critical strengths
-
The paper studies a possibly interesting theoretical problem: whether preserving current predictions is sufficient for future learning. Given the importance of memory in machine learning, this work could be of interest to a broad range of readers.
-
The authors provide several theoretical results covering different learning settings, including ridge regression, logistic regression, and attention mechanisms. These results could provide useful insights into the memory requirements of learning systems.
Critical weaknesses
- Poor writing and unclear research motivation.
My main concern is that the paper is very difficult to follow. Even after carefully reading the Introduction, I still find it hard to understand the background of this research and why this problem is important. The paper is written in a very theoretical and abstract way, with many mathematical concepts introduced without sufficient intuitive explanations. This makes it difficult for readers to understand the main contributions and judge the value of this work. I think the authors should substantially improve the presentation, especially the Introduction, by providing clearer motivation, more intuitive examples, and a better explanation of the research problem.
- Unclear novelty and practical significance.
Although the paper provides several theoretical results, it is not clear to me what the main new insight is compared with existing work on sufficient statistics, online learning, and memory representations. The paper also covers many different settings, including regression, logistic regression, and attention, but their connections and practical implications are not sufficiently explained. Moreover, the work is almost entirely theoretical, without empirical evaluation on practical learning tasks. While I understand that experiments are not always necessary for theoretical papers, some simple experiments or concrete examples would help readers better understand the significance of the theoretical findings.
Questions for authors
-
Could the authors explain more clearly what problem this paper is trying to address and why it matters? I found it difficult to understand the main motivation, even after carefully reading the Introduction.
-
What is the main technical novelty of this work compared with previous studies on sufficient statistics, online learning, and memory representations? What are the key new insights that distinguish this paper from existing work?
-
Could the authors explain how these theoretical findings relate to practical learning systems? For example, are there any concrete cases where the limitations identified in this paper actually affect model performance?
Thank you for requesting clearer motivation, precise novelty claims and concrete examples. The revision adds an explicit reference task, a definitions section and an executed synthetic comparison. We address your three questions directly.
We ask what a compressed history must retain to reproduce the predictor obtained by fitting the entire accumulated history, including after another observation arrives. The crucial issue is that zero error on every current prediction does not certify that the retained state can reproduce the next update.
The introduction demonstrates this with scalar ridge regression: Fh(w)=21w2+21∑(x,y)∈h(xw−y)2. Consider one zero-label observation with feature 1 or 2. The current predictors agree, but after the same observation (1,1), wA=wB=0,wA+=31,wB+=61. The second history supplies stronger evidence supporting zero, so the new observation should move its predictor less. That evidence is absent from the current prediction function.
Section 2 defines the common setting and distinguishes reproducing the cumulative minimizer from statistics sufficient for online regret guarantees (Foster et al., 2018). The question is which information must survive compression to support the intended future learning computation.
Section 4.2 explicitly acknowledges classical sufficiency, including Koopman, Hipp and Mattner. Our contribution is the operational characterization and its sharp consequences:
Information forced by allowed updates. Theorem 2 proves that equality of optimizer responses to every possible ordinary single additional observation is equivalent to equality of the entire accumulated gradient function. Under the stated unrestricted observation domain, it establishes exact state dimensions using unit-weight histories, continuous encoders and unrestricted decoders. Theorem 3 quantifies what remains necessary after fixing the current predictor. These are necessity statements for every admissible representation.
A bounded obstruction to finite derivative summaries. Theorems 4–5 construct scalar logistic histories with bounded features, the same current predictor and the same prescribed finite list of objective derivatives, while their possible continuation responses still distinguish them. A fixed amount of local curvature information therefore does not generally determine exact cumulative logistic updates.
A finite-precision tradeoff. Theorem 6 gives matching average-error bounds as a function of stored bits on a conditioned ridge family whose current predictors all agree. Compression precedes the unknown future update and query; the lower bound permits arbitrary randomized encoders and unrestricted decoders.
Ridge exposes the missing evidence; general residual learning identifies it; bounded logistic learning tests whether local derivatives suffice; attention gives the parallel distinction between normalized predictions and the numerator and mass needed for continuation. Table 1 summarizes these connections.
Yes. Figure 1 and Section 8 report a comparison over 41 ridge histories with the same zero current predictor and the same diagonal evidence, but different off-diagonal evidence. After one shared observation, full recursive least squares matches the full-history batch solution numerically. The tested diagonal-evidence and raw delta recurrences return zero at the diagnostic query. At the two endpoint histories,
f_{\mathrm{diagonal}}=f_{\mathrm{delta}}=0,\qquad \mathrm{MSE}_{\mathrm{endpoints}}=4/49.$$ A test comparing only current predictions would score all these states perfectly. The common continuation exposes the lost information and its effect on prediction. The experiment evaluates the specified recurrences. Learned channels, auxiliary states or replay may preserve additional evidence and must enter the memory accounting. The approximation question is also addressed. The logistic theorem concerns exact continuation, while Propositions 9–10 bound future error from approximation of the gradient function or attention numerator and mass. This gives a constructive criterion: approximate the evidence needed by the intended future computation, and assess the state after shared updates as well as before them.