- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies Memory Consolidation: internalizing knowledge or behavioral changes induced by context into model parameters while preserving existing capabilities. Cortex constructs rehearsal queries and responses from context, then distills a context-conditioned teacher into FFN updates through LoRA or structural expansion. The authors evaluate consolidation and retention across knowledge editing, API learning, agent experience, persona steering, and knowledge suppression, using several dense and MoE models.
Critical strengths
The problem formulation is well connected to the evaluation. I find it useful to study these different tasks through the common objectives of consolidation and retention. The paper makes concrete what should be learned from context and what should be preserved in each setting.
The experiments are extensive and substantiate the main design choices. Beyond comparisons across tasks and model architectures, the paper includes controls using the same training data and analyses of FFN update location, replay, hidden-state alignment, and shared expansion in MoE models. I appreciate the effort to understand which choices contribute to learning and retention.
Critical weaknesses
The evaluation of continual consolidation is narrower than the rest of the experimental study. It uses a fixed sequence of 50 factual edits on one model, leaving uncertainty about longer update sequences and sequences involving different kinds of context. Studying the relationship between accumulating capacity and retention in these settings would be a useful extension.
Related work is generally well covered. However, the connections to synthetic-data-based knowledge learning and modular knowledge storage could be strengthened. Discussing Lampinen et al., New News, Learning Facts at Scale with Active Reading, and Back et al. would help connect Cortex’s prospective-query generation and parametric memory design to this literature.
Questions for authors
Under what conditions is Structural expansion preferable to Synaptic updates? I would be interested in the authors’ guidance based on the results across tasks and models. What factors appear to determine their relative strengths, taking the capacity ablations into account?
How does continual consolidation balance preserving existing memories with revising them when new context changes previously learned information? The paper aims to preserve earlier “still-valid memories,” while query replay encourages preservation of the previous model’s responses. How does the current procedure handle cases where part of an earlier memory is no longer valid?
In Appendix F.1, is the “held-out evaluation split” used for baseline hyperparameter selection separate from the evaluation set used to report the final results? Please clarify this distinction.
5 comments
Response to Reviewer GYfj (Part 2/3, Question 1 and Question 2)
Response to Question1: Choosing Structural or Synaptic
Under what conditions is Structural expansion preferable to Synaptic updates? I would be interested in the authors’ guidance based on the results across tasks and models. What factors appear to determine their relative strengths, taking the capacity ablations into account?
Summary: For single or batch factual corrections we would start with Synaptic, which had higher CR in seven of eight comparisons. For new APIs or a sequence of accumulating edits we would start with Structural, which had higher CR in six of eight API comparisons and higher final CR and RR in both edit orders. Both variants worked well on steering (§5.1, Appendix H, and §5.2, Table 2).
-
Revising an existing association. Synaptic changes the effective mapping of an existing FFN projection through a low-rank update. This may help when an existing question calls for a revised answer (§5.1, lines 317–321).
-
Accumulating knowledge. Structural adds nonlinear channels. Both variants freeze earlier writes, but later updates can still interfere with earlier behavior. Without replay, after 25 subsequent edits, mean accuracy on original and rephrased questions for the first 25 facts was 40.5% with Structural and 8.0% with Synaptic (Fig. 5e). In the capacity sweep, a single new channel per edit already gave Structural its highest mean CR (Fig. 5m,n).
-
When target recovery is similar. Retention can distinguish configurations with similar CR. On GPT-OSS-20B ALFWorld, Structural reached 97% CR and 95% RR. Synaptic reached 101% CR and 23% RR (Fig. 3o).
Response to Question2: Revising an earlier memory
How does continual consolidation balance preserving existing memories with revising them when new context changes previously learned information? The paper aims to preserve earlier “still-valid memories,” while query replay encourages preservation of the previous model’s responses. How does the current procedure handle cases where part of an earlier memory is no longer valid?
Summary: Replay can preserve an answer after it has become outdated. In a four-country succession experiment, adding validity years to the training facts and questions and supplying the current date at recall raised final current-president accuracy from 15.3 ± 2.4% to 86.1 ± 20.6%. All four capitals remained correct at every update (Appendix F.6, p. 40).
-
What to preserve when a fact changes. Each fictional country changed president twice. In the dated condition, the source facts and rehearsal questions specified the year of each term, and the system prompt supplied the current date at evaluation. Both conditions used the same undated user questions and the same update and replay schedules. After each update we asked about the current president and the unchanged capital. Recall of former presidents by year remains untested. Appendix F.6 reports the final update. The table gives all three for Structural (k=16), as mean ± sample SD over three seeds.
Accuracy (%) President after update 1 President after update 2 President after update 3 Capital at every update Without dates 100.0 ± 0.0 54.2 ± 7.2 15.3 ± 2.4 100.0 ± 0.0 With validity years and current date 100.0 ± 0.0 70.8 ± 31.5 86.1 ± 20.6 100.0 ± 0.0 In LoCoMo, the question generator converted relative dates to absolute dates when the session supplied enough information. In the 50-edit protocol, every current example enters its update, while only a subset of old questions is used once the pool exceeds the 50-question replay limit (Table 5). This favors exposure to each current example but cannot tell the model when an old answer ceased to apply.
-
When the previous answer was wrong from the start. Here the relevant evidence is the corrected source. Cortex distills the answer its frozen teacher generates from that source. If the teacher follows the correction, the student receives the revised answer. If the teacher relies on its prior belief, it can reproduce the original error, a kind of context–knowledge conflict also observed by Longpre et al. [6].
Response to Reviewer GYfj (Part 3/3, Question 3)
Response to Question3: Tuning data and final evaluation
In Appendix F.1, is the “held-out evaluation split” used for baseline hyperparameter selection separate from the evaluation set used to report the final results? Please clarify this distinction.
Summary: The tuning and final-evaluation items are separate for ALFWorld, τ²-Bench, Unlearn, Code, and Steering. Editing uses the requested target question for both selection and the reported efficacy score. We now spell out that distinction in Appendix F.1 (p. 31, lines 1663–1673).
We selected hyperparameters using target-side performance only; retention rate (RR) did not enter selection. Tuning and final evaluation used separate questions or episodes for every task except editing efficacy, where the target question is part of the edit request itself. The table gives the split for each task (Appendix F.1, p. 31, lines 1663–1673; evaluation counts in Tables 4, 7, and 15).
| Task | Data used to select hyperparameters | Final evaluation |
|---|---|---|
| ALFWorld | 30% of the training pool for tuning; remaining 70% for training | Separate evaluation episodes |
| τ²-Bench | 20% of the training pool for tuning; remaining 80% for training | Separate evaluation tasks |
| Factual editing | Efficacy on the target question supplied with each edit | Efficacy on the same question; rephrasing, portability, and locality scored separately and excluded from selection |
| Unlearn | 10 manually constructed tuning questions per target person | Separate evaluation questions; tuning questions excluded from training and final evaluation |
| Code | About 5 manually constructed tuning questions per document | Separate evaluation questions; tuning questions excluded from training and final evaluation |
| Steering | 20 manually constructed tuning questions | Separate sets of 200 Power-seeking and 1,000 Psychopathy questions |
During selection, we used three generations per editing query and one generation or environment episode per item for the other tasks. We retrained task-trained baselines with the selected configuration before final evaluation. The editing definition of efficacy follows Meng et al. [7]; using it for hyperparameter selection was our protocol.
References
- Maharana et al. (2024). ACL. Evaluating Very Long-Term Conversational Memory of LLM Agents.
- Lampinen et al. (2025). NeurIPS. On the Generalization of Language Models from In-Context Learning and Finetuning: A Controlled Study.
- Park et al. (2025). Arxiv. New News: System-2 Fine-tuning for Robust Integration of New Knowledge.
- Lin et al. (2025). ICLR. Learning Facts at Scale with Active Reading.
- Back et al. (2026). ICML. Understanding LoRA as Knowledge Memory: An Empirical Analysis.
- Longpre et al. (2021). EMNLP. Entity-Based Knowledge Conflicts in Question Answering.
- Meng et al. (2022). NeurIPS. Locating and Editing Factual Associations in GPT.
Response to Reviewer GYfj (Part 1/3, Weakness 1 and Weakness 2)
Dear Reviewer GYfj,
Thank you for highlighting how the consolidation and retention objectives make clear what each task asks the model to learn and what it should preserve. We also appreciate your assessment of the controlled experiments, particularly the comparisons using the same training data and those testing FFN versus attention updates.
We have highlighted the revisions in green in the manuscript and respond to your questions and suggestions below.
Response to Weakness1: History length, heterogeneous updates, and capacity
Summary: We tested longer conversational histories and a sequence that changes task type. Across two LoCoMo conversations, increasing Structural’s width fourfold produced no consistent gain in final F1. In a separate three-task sequence, editing CR stayed unchanged after the steering update on both Qwen2.5-14B and GPT-OSS-20B. The longest sequence by update count remains the 50-edit experiment (Appendices F.5–F.6, Tables 12–13).
Longer histories and accumulated capacity. On LoCoMo [1], we updated Qwen2.5-14B once per session over 19 sessions (15K tokens) on conv-30 and 32 sessions (29K tokens) on conv-41. The histories span roughly six and eight months. Only the full-dialogue reference saw the conversation when answering the held-out questions. Table 12 reports both Structural widths alongside the two references, Locas-GLU trained with next-token prediction, and Synaptic.
F1 is the mean ± sample SD over three training seeds for adapted models and three sampling runs for the references. CR uses each dialogue’s vanilla and full-dialogue mean F1 (Eq. 3). Locas-GLU matched the final capacity of Structural at k=16. Increasing k from 4 to 16 raised the cumulative added weights from 56.0M to 224.1M on conv-30 and from 94.4M to 377.5M on conv-41. Final F1 changed by −0.42 points on conv-30 and +1.01 on conv-41. For the 50-edit sequence, Fig. 5n shows how non-target retention varies with added width.
Different contexts and backbones. We successively consolidated API documentation, factual edits, and a steering task. On Qwen2.5-14B, API CR rose from 61.7% to 66.6% after editing and returned to 61.7% after steering. On OSS-20B, it fell from 85.8% to 82.0% after editing and stayed there after steering. Editing CR stayed at 90.5% and 76.2%, respectively, after steering (Table 13).
Response to Weakness2: Related works
Summary: We added the suggested connections where we introduce question generation (§3.1) and where we compare forms of parametric memory (Appendix C, p. 24). The first group of papers helps explain how to choose learning examples from a source. The modular-memory comparison helps explain where Cortex stores the resulting update.
Generating useful examples. Lampinen et al. [2] examine how reasoning traces affect generalization from learned facts. New News [3] and Active Reading [4] show how questions or other study materials can be generated from source documents. In Cortex, the question also identifies an occasion on which the student should reproduce the behavior of the context-conditioned teacher.
Choosing a parameterization. Back et al. [5] study the capacity and composition of modular LoRA memories. The closest Cortex variant is Synaptic, which restricts the low-rank update to FFN projections. Structural adds nonlinear FFN channels instead. Appendix C discusses the relationship between these choices and what the model needs to recall from its parameters.