- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies when and how knowledge should be consolidated from external memory into model parameters. It formulates memory consolidation as an online rent-or-buy problem and provides theoretical results on promotion decisions, interference costs, and memory allocation. The analysis offers useful insights into the trade-offs involved in knowledge consolidation.
Critical strengths
- This work addresses an important question for pretrained models: when and how should external knowledge be consolidated into model parameters?
- This paper is clearly written and logically structured. The theoretical analysis is presented in a coherent manner.
Critical weaknesses
- The theoretical analysis primarily focuses on Gaussian classifiers with frozen features rather than sequential models. In sequential models, such as LLMs, knowledge interference can affect subsequent generation steps and lead to more complex interactions. It remains unclear whether the theoretical conclusions extend to these settings.
- The theoretical framework allows consolidated knowledge to be demoted at a specified cost. However, demotion in pretrained models may require model editing or unlearning, which can introduce additional interference and affect previously acquired knowledge. These effects are not explicitly modeled, limiting the practical applicability of the theoretical guarantees.
- The analysis relies on idealized assumptions, including Gaussian class distributions, shared covariance, and frozen feature representations. These assumptions may not hold in practical continual learning systems. Although the paper evaluates some deviations from these assumptions, the generalizability of the derived results to more complex models remains unclear.
Questions for authors
Some knowledge may become outdated or invalid as the environment changes. In such cases, previously consolidated knowledge may no longer be beneficial and could even become harmful. How does the proposed framework account for changes in knowledge validity over time, beyond changes in query frequency? Can the framework support updating or replacing outdated consolidated knowledge?
Thank you for the long reply. It basically covers my questions: the full list of the six lost tests, the F-7 explanation under the registered rule, the plan to redraw Figure 1b, I accept these. The larger experiments, the iCaRL buffer arm, the new embedder, the Pythia-160M run, are all planned for the revision. I will value them when I see them, but they were not part of the submission I reviewed. This is why my score moves only a little.
My main remaining problem is the readability. The paper borrows its core concepts from another field, rent-or-buy, competitive ratios, crowding, and applies them to memory consolidation. Borrowing ideas is fine, but then the paper owes the reader one framework figure that shows the whole setting in a single picture, who pays rent, who pays the one-time price, where crowding enters the cost. There is no such figure, so the reader has to assemble the model from text and formulas. The writing is also not academic enough, especially the style of the headings, and with so many formulas the main innovation, the contrast between Theorem 1 and Theorem 2, does not stand out, I had to dig for it.
One small thing: in Figure 2c the fonts of the axis labels are inconsistent with the other panels. This is simple, easy to find and easy to fix, exactly the kind of thing a careful pass should catch. Please also unify the figure and table styles in the revision.
With the rebuttal covering most of my questions I raise my score from 1 to 2. At the same time I lower my presentation score from 2 to 1. These two moves go together: the rating goes up because my questions were answered, the presentation goes down because, rereading the paper for this discussion, it is still difficult to follow, for the reasons above. My hope is that the authors revise the manuscript to make it easier to understand, with a framework figure and consistent figures.