acceptodds
Under review as a conference paper at ICLR 2027

Cortex: Consolidating Context into Parametric Memory

Abstract

Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B–120B models, dense and mixture-of-experts architectures, and text-only and multimodal settings, Cortex lies on the empirical consolidation–retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance. We will release the testbed and open-source Cortex.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory, Continual Learning, Model Edit, Unlearn, Steering, Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GYfjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies Memory Consolidation: internalizing knowledge or behavioral changes induced by context into model parameters while preserving existing capabilities. Cortex constructs rehearsal queries and responses from context, then distills a context-conditioned teacher into FFN updates through LoRA or structural expansion. The authors evaluate consolidation and retention across knowledge editing, API learning, agent experience, persona steering, and knowledge suppression, using several dense and MoE models.

Critical strengths

The problem formulation is well connected to the evaluation. I find it useful to study these different tasks through the common objectives of consolidation and retention. The paper makes concrete what should be learned from context and what should be preserved in each setting.

The experiments are extensive and substantiate the main design choices. Beyond comparisons across tasks and model architectures, the paper includes controls using the same training data and analyses of FFN update location, replay, hidden-state alignment, and shared expansion in MoE models. I appreciate the effort to understand which choices contribute to learning and retention.

Critical weaknesses

The evaluation of continual consolidation is narrower than the rest of the experimental study. It uses a fixed sequence of 50 factual edits on one model, leaving uncertainty about longer update sequences and sequences involving different kinds of context. Studying the relationship between accumulating capacity and retention in these settings would be a useful extension.

Related work is generally well covered. However, the connections to synthetic-data-based knowledge learning and modular knowledge storage could be strengthened. Discussing Lampinen et al., New News, Learning Facts at Scale with Active Reading, and Back et al. would help connect Cortex’s prospective-query generation and parametric memory design to this literature.

Questions for authors

Under what conditions is Structural expansion preferable to Synaptic updates? I would be interested in the authors’ guidance based on the results across tasks and models. What factors appear to determine their relative strengths, taking the capacity ablations into account?

How does continual consolidation balance preserving existing memories with revising them when new context changes previously learned information? The paper aims to preserve earlier “still-valid memories,” while query replay encourages preservation of the previous model’s responses. How does the current procedure handle cases where part of an earlier memory is no longer valid?

In Appendix F.1, is the “held-out evaluation split” used for baseline hyperparameter selection separate from the evaluation set used to report the final results? Please clarify this distinction.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GYfj (Part 1/3, Weakness 1 and Weakness 2)

Dear Reviewer GYfj,

Thank you for highlighting how the consolidation and retention objectives make clear what each task asks the model to learn and what it should preserve. We also appreciate your assessment of the controlled experiments, particularly the comparisons using the same training data and those testing FFN versus attention updates.

We have highlighted the revisions in green in the manuscript and respond to your questions and suggestions below.

Response to Weakness1: History length, heterogeneous updates, and capacity

The evaluation of continual consolidation is narrower than the rest of the experimental study. It uses a fixed sequence of 50 factual edits on one model, leaving uncertainty about longer update sequences and sequences involving different kinds of context. Studying the relationship between accumulating capacity and retention in these settings would be a useful extension.

Summary: We tested longer conversational histories and a sequence that changes task type. Across two LoCoMo conversations, increasing Structural’s width fourfold produced no consistent gain in final F1. In a separate three-task sequence, editing CR stayed unchanged after the steering update on both Qwen2.5-14B and GPT-OSS-20B. The longest sequence by update count remains the 50-edit experiment (Appendices F.5–F.6, Tables 12–13).

  1. Longer histories and accumulated capacity. On LoCoMo [1], we updated Qwen2.5-14B once per session over 19 sessions (15K tokens) on conv-30 and 32 sessions (29K tokens) on conv-41. The histories span roughly six and eight months. Only the full-dialogue reference saw the conversation when answering the held-out questions. Table 12 reports both Structural widths alongside the two references, Locas-GLU trained with next-token prediction, and Synaptic.

    Dialogue / metricVanillaFull dialogue in contextLocas-GLUSynaptic (r=4)Structural (k=4)Structural (k=16)
    conv-30 / F1 (%)0.00 ± 0.0022.57 ± 0.222.78 ± 0.0018.72 ± 1.9021.41 ± 4.4320.99 ± 1.00
    conv-30 / CR (%)0.0100.012.382.994.993.0
    conv-41 / F1 (%)0.21 ± 0.0018.87 ± 0.071.25 ± 0.3017.34 ± 0.5219.18 ± 2.5920.19 ± 0.96
    conv-41 / CR (%)0.0100.05.691.8101.7107.1

F1 is the mean ± sample SD over three training seeds for adapted models and three sampling runs for the references. CR uses each dialogue’s vanilla and full-dialogue mean F1 (Eq. 3). Locas-GLU matched the final capacity of Structural at k=16. Increasing k from 4 to 16 raised the cumulative added weights from 56.0M to 224.1M on conv-30 and from 94.4M to 377.5M on conv-41. Final F1 changed by −0.42 points on conv-30 and +1.01 on conv-41. For the 50-edit sequence, Fig. 5n shows how non-target retention varies with added width.

  1. Different contexts and backbones. We successively consolidated API documentation, factual edits, and a steering task. On Qwen2.5-14B, API CR rose from 61.7% to 66.6% after editing and returned to 61.7% after steering. On OSS-20B, it fell from 85.8% to 82.0% after editing and stayed there after steering. Editing CR stayed at 90.5% and 76.2%, respectively, after steering (Table 13).

    BackboneAPI CR (%) after stages 1 → 2 → 3Editing CR (%) after stages 2 → 3Steering CR (%) at stage 3
    Qwen2.5-14B61.7 → 66.6 → 61.790.5 → 90.5100.4
    GPT-OSS-20B85.8 → 82.0 → 82.076.2 → 76.2100.2

Response to Weakness2: Related works

Related work is generally well covered. However, the connections to synthetic-data-based knowledge learning and modular knowledge storage could be strengthened. Discussing ...(some works) would help connect Cortex’s prospective-query generation and parametric memory design to this literature.

Summary: We added the suggested connections where we introduce question generation (§3.1) and where we compare forms of parametric memory (Appendix C, p. 24). The first group of papers helps explain how to choose learning examples from a source. The modular-memory comparison helps explain where Cortex stores the resulting update.

  1. Generating useful examples. Lampinen et al. [2] examine how reasoning traces affect generalization from learned facts. New News [3] and Active Reading [4] show how questions or other study materials can be generated from source documents. In Cortex, the question also identifies an occasion on which the student should reproduce the behavior of the context-conditioned teacher.

  2. Choosing a parameterization. Back et al. [5] study the capacity and composition of modular LoRA memories. The closest Cortex variant is Synaptic, which restricts the low-rank update to FFN projections. Structural adds nonlinear FFN channels instead. Appendix C discusses the relationship between these choices and what the model needs to recall from its parameters.

AuthorsOpenReview ↗

Response to Reviewer GYfj (Part 2/3, Question 1 and Question 2)

Response to Question1: Choosing Structural or Synaptic

Under what conditions is Structural expansion preferable to Synaptic updates? I would be interested in the authors’ guidance based on the results across tasks and models. What factors appear to determine their relative strengths, taking the capacity ablations into account?

Summary: For single or batch factual corrections we would start with Synaptic, which had higher CR in seven of eight comparisons. For new APIs or a sequence of accumulating edits we would start with Structural, which had higher CR in six of eight API comparisons and higher final CR and RR in both edit orders. Both variants worked well on steering (§5.1, Appendix H, and §5.2, Table 2).

  1. Revising an existing association. Synaptic changes the effective mapping of an existing FFN projection through a low-rank update. This may help when an existing question calls for a revised answer (§5.1, lines 317–321).

  2. Accumulating knowledge. Structural adds nonlinear channels. Both variants freeze earlier writes, but later updates can still interfere with earlier behavior. Without replay, after 25 subsequent edits, mean accuracy on original and rephrased questions for the first 25 facts was 40.5% with Structural and 8.0% with Synaptic (Fig. 5e). In the capacity sweep, a single new channel per edit already gave Structural its highest mean CR (Fig. 5m,n).

  3. When target recovery is similar. Retention can distinguish configurations with similar CR. On GPT-OSS-20B ALFWorld, Structural reached 97% CR and 95% RR. Synaptic reached 101% CR and 23% RR (Fig. 3o).


Response to Question2: Revising an earlier memory

How does continual consolidation balance preserving existing memories with revising them when new context changes previously learned information? The paper aims to preserve earlier “still-valid memories,” while query replay encourages preservation of the previous model’s responses. How does the current procedure handle cases where part of an earlier memory is no longer valid?

Summary: Replay can preserve an answer after it has become outdated. In a four-country succession experiment, adding validity years to the training facts and questions and supplying the current date at recall raised final current-president accuracy from 15.3 ± 2.4% to 86.1 ± 20.6%. All four capitals remained correct at every update (Appendix F.6, p. 40).

  1. What to preserve when a fact changes. Each fictional country changed president twice. In the dated condition, the source facts and rehearsal questions specified the year of each term, and the system prompt supplied the current date at evaluation. Both conditions used the same undated user questions and the same update and replay schedules. After each update we asked about the current president and the unchanged capital. Recall of former presidents by year remains untested. Appendix F.6 reports the final update. The table gives all three for Structural (k=16), as mean ± sample SD over three seeds.

    Accuracy (%)President after update 1President after update 2President after update 3Capital at every update
    Without dates100.0 ± 0.054.2 ± 7.215.3 ± 2.4100.0 ± 0.0
    With validity years and current date100.0 ± 0.070.8 ± 31.586.1 ± 20.6100.0 ± 0.0

    In LoCoMo, the question generator converted relative dates to absolute dates when the session supplied enough information. In the 50-edit protocol, every current example enters its update, while only a subset of old questions is used once the pool exceeds the 50-question replay limit (Table 5). This favors exposure to each current example but cannot tell the model when an old answer ceased to apply.

  2. When the previous answer was wrong from the start. Here the relevant evidence is the corrected source. Cortex distills the answer its frozen teacher generates from that source. If the teacher follows the correction, the student receives the revised answer. If the teacher relies on its prior belief, it can reproduce the original error, a kind of context–knowledge conflict also observed by Longpre et al. [6].

AuthorsOpenReview ↗

Response to Reviewer GYfj (Part 3/3, Question 3)

Response to Question3: Tuning data and final evaluation

In Appendix F.1, is the “held-out evaluation split” used for baseline hyperparameter selection separate from the evaluation set used to report the final results? Please clarify this distinction.

Summary: The tuning and final-evaluation items are separate for ALFWorld, τ²-Bench, Unlearn, Code, and Steering. Editing uses the requested target question for both selection and the reported efficacy score. We now spell out that distinction in Appendix F.1 (p. 31, lines 1663–1673).

We selected hyperparameters using target-side performance only; retention rate (RR) did not enter selection. Tuning and final evaluation used separate questions or episodes for every task except editing efficacy, where the target question is part of the edit request itself. The table gives the split for each task (Appendix F.1, p. 31, lines 1663–1673; evaluation counts in Tables 4, 7, and 15).

TaskData used to select hyperparametersFinal evaluation
ALFWorld30% of the training pool for tuning; remaining 70% for trainingSeparate evaluation episodes
τ²-Bench20% of the training pool for tuning; remaining 80% for trainingSeparate evaluation tasks
Factual editingEfficacy on the target question supplied with each editEfficacy on the same question; rephrasing, portability, and locality scored separately and excluded from selection
Unlearn10 manually constructed tuning questions per target personSeparate evaluation questions; tuning questions excluded from training and final evaluation
CodeAbout 5 manually constructed tuning questions per documentSeparate evaluation questions; tuning questions excluded from training and final evaluation
Steering20 manually constructed tuning questionsSeparate sets of 200 Power-seeking and 1,000 Psychopathy questions

During selection, we used three generations per editing query and one generation or environment episode per item for the other tasks. We retrained task-trained baselines with the selected configuration before final evaluation. The editing definition of efficacy follows Meng et al. [7]; using it for hyperparameter selection was our protocol.

References

  1. Maharana et al. (2024). ACL. Evaluating Very Long-Term Conversational Memory of LLM Agents.
  2. Lampinen et al. (2025). NeurIPS. On the Generalization of Language Models from In-Context Learning and Finetuning: A Controlled Study.
  3. Park et al. (2025). Arxiv. New News: System-2 Fine-tuning for Robust Integration of New Knowledge.
  4. Lin et al. (2025). ICLR. Learning Facts at Scale with Active Reading.
  5. Back et al. (2026). ICML. Understanding LoRA as Knowledge Memory: An Empirical Analysis.
  6. Longpre et al. (2021). EMNLP. Entity-Based Knowledge Conflicts in Question Answering.
  7. Meng et al. (2022). NeurIPS. Locating and Editing Factual Associations in GPT.
Reviewer_fqPoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the question how to integrate context's general knowledge into the model so that the model performs well for the context domain but not degrade for general domain performance.

To answer this question, the paper has proposed a benchmark as well as its own method. The paper's method consists of a post-training method (with anticipated query, answer as training signals), modify targeted LLM modules and train to mimic the teacher behaviour without context.

Empirically, the authors show that their proposed method offers an interesting Pareto-front compared to existing literature across different LLMs. The ablation study mainly confirms the contribution of the targeted module design as well as the training method.

Critical strengths

The paper has proposed tests on various benchmarks, across different LLMs and compared with various existing methods. The paper show in Figure 1 that the proposed method on both IID and OOD situations indeed offer an interesting Pareto Front compared to existing methods (which generally are low CR or low RR in papers' terms).

The proposed methology are not novel in itself but the paper has its originality seemingly to combine these into the proposed pipeline which shows some empirical success.

Critical weaknesses

In terms of formatting, the paper mentions few related work in its main sections. The main table does compare with existing works (and arguably extensively) but without discussion and Figure 1 is hard to read or get at first view.

The ablation on the data construction seems weak while it seems to be an important contribution made by the paper.

Questions for authors

Distilling knowledge into LLMs models the way as the paper used have shown a major risk that increases the hallucination, has the paper investigated into this area?

4 comments
AuthorsOpenReview ↗

Response to Reviewer fqPo (Part 1/2, Weakness 1 and Weakness 2)

Dear Reviewer fqPo,

Thank you for highlighting the comparisons across benchmarks, models, and baselines and the trade-off between consolidation and retention visible across these settings. We also appreciate your observation that the ablations helped explain how the choice of which modules to update and how to train them contributed to the results.

We have highlighted the revisions in green in the manuscript and respond to your questions and suggestions below.

Response to Weakness1: Reading the figures and interpreting the baselines

In terms of formatting, the paper mentions little related work in its main sections. The main table does compare with existing works (and arguably extensively) but without discussion, Figure 1 is hard to read or get at first view.

Summary: We added a method-level analysis to §5.1 and Appendix F.2, explained the axes in the caption of Fig. 3, and connected prospective question generation to earlier work in §3.1 and Appendix C. Locas and δ-Mem preserved non-target performance while learning little of the new APIs. Doc2LoRA recovered more of the context-induced gain when we wrote each fact separately.

  1. Reading Figs. 1 and 3. Figure 1(b) shows what changes between learning and recall. During learning, the teacher sees the source context. At recall, the updated model gets the new question and uses the changes stored in the highlighted FFN. The Pareto plots in Fig. 3 compare methods on consolidation and retention. Moving right means recovering more of the benefit of seeing the source (CR), and moving up means preserving more performance on unrelated tasks (RR). For example, in Fig. 3(o), Structural and Synaptic both recover almost all of the target gain on ALFWorld (97% and 101% CR), while their retention differs substantially (95% and 23% RR). The revised Fig. 3 caption explains both axes (§5.1, p. 7).

  2. What the baselines show. In our batch API evaluation, Locas [1] and δ-Mem (SSW) [2] retained 98–101% RR on the two Qwen backbones yet recovered at most 11.4% of the benefit of seeing the documentation. For Doc2LoRA [3] on Qwen3-4B-Instruct, CR fell from 95% with individual facts to 24% with the same 50 facts written jointly. Joint editing changes both source length and the number of facts one update must encode (§5.1, pp. 6–7, and Appendix F.2, p. 33).

  3. Connection to prior work. Section 3.1 now connects rehearsal queries to Self-QA in New News [4] and learning from generated study material in Active Reading [5]. Appendix C explains how Cortex uses these queries to train the student to reproduce what the teacher can do with the source.

Response to Weakness2: Sensitivity to question construction

The ablation on the data construction seems weak while it seems to be an important contribution made by the paper.

Summary: We compared generic and task-specific question generation in two additional experiments for this response. In both conditions, the generator received the source material and had no access to held-out evaluation questions or answers. Grounding dialogue questions in speakers and dates improved recall by 12.23 F1 points on LoCoMo conv-30. Generating questions that exercise a documented API improved batch API learning by 21.7 CR points.

Qwen2.5-14B StructuralGeneric questionsTask-specific questionsGain (points)
LoCoMo conv-30, k=1; token F1 (%)8.7620.99+12.23
Batch API learning, k=128; CR (%)52.073.7+21.7

The dialogue comparison used the same width and three training seeds. On conv-30, the F1 gain from 8.76 to 20.99 corresponds to a rise in CR from 38.8% to 93.0%, using the vanilla and full-dialogue mean F1 scores of 0.00 and 22.57 (Eq. 3; Table 12). The API comparison held training settings and question budget fixed. Table 12 reports the across-seed SD for task-specific LoCoMo. The standard API result appears in Table 18. The additional generic-prompt comparisons are reported here. For both tasks, specifying what the model would need to do with the source improved the downstream score.

AuthorsOpenReview ↗

Response to Reviewer fqPo (Part 2/2, Question 1)

Response to Question1: Factual errors after consolidation

Distilling knowledge into LLMs models the way as the paper used have shown a major risk that increases the hallucination, has the paper investigated into this area?

Summary: A factual error can enter a rehearsal example during writing or appear when the updated model answers a new question. For LoCoMo, we checked generated training pairs against the source session. After 50 factual edits, we examined both overall retention and individual answers to unrelated SimpleQA questions. Both Cortex variants gave incorrect answers more often after editing.

  1. During writing. A generated question can presuppose an event that never happened, and a teacher answer can introduce a detail absent from the source. In LoCoMo, we kept a generated pair only when the checker judged that the current session supported its answer (Appendix F.5).

  2. After writing. We evaluated Qwen2.5-14B-Instruct after the original sequence of 50 factual edits at two levels.

    • Overall retention. RR compares the equal-weight mean of locality, MMLU, SimpleQA accuracy, and GSM8K after editing with the same mean before editing. Final RR was 99.8% for Structural and 97.2% for Synaptic (Table 21). This aggregate tracks how much non-target performance is retained. Its SimpleQA component gives zero credit to both an incorrect answer and a non-answer.
    • Factual answers on SimpleQA. On the same 1,000 unrelated questions [6], the incorrect-answer rate rose from 72.90% before editing to 93.90% for Structural and 95.73% for Synaptic. The share graded “not attempted” fell from 22.50% to 1.30% and 0.37%, respectively. Accuracy changed from 4.60% to 4.80% and 3.90%. The models attempted more questions after editing, with little change in the share answered correctly.

    Table 7 gives the evaluation sample size, and Table 21 reports SimpleQA accuracy. We also computed F-score from the same judgments as the harmonic mean of overall accuracy and accuracy among attempted questions [6]. Edited entries below report mean ± sample SD across three runs.

    SimpleQA after 50 edits (%)UneditedROMEMEMITAlphaEditWISEFT-MStructural (k=1)Synaptic (r=1)
    Accuracy4.603.60 ± 0.003.80 ± 0.463.17 ± 0.153.80 ± 0.443.57 ± 0.424.80 ± 0.173.90 ± 0.56
    F-score5.184.48 ± 0.074.70 ± 0.543.91 ± 0.184.13 ± 0.534.36 ± 0.514.83 ± 0.183.91 ± 0.56

    Table 20 gives a separate comparison with the distillation baseline SDCD on SimpleQA accuracy after all 50 facts were written jointly.

References

  1. Lu et al. (2026). Arxiv. Locas: Your Models are Principled Initializers of Locally-Supported Parametric.
  2. Lei et al. (2026). NeurIPS. δ-mem: Efficient Online Memory for Large Language Models.
  3. Charakorn et al. (2026). PMLR. Doc-to-LoRA: Learning to Instantly Internalize Contexts.
  4. Park et al. (2025). Arxiv. New News: System-2 Fine-tuning for Robust Integration of New Knowledge.
  5. Lin et al. (2025). Arxiv. Learning Facts at Scale with Active Reading.
  6. Wei et al. (2024). Arxiv. Measuring Short-Form Factuality in Large Language Models.

The authors have properly addressed my questions on the paper that hopefully will strengthen the paper. I don't have further concerns and my score remains.

AuthorsOpenReview ↗

Thank you for your feedback and support

Dear Reviewer fqPo,

Thank you for reviewing our responses and confirming that your concerns have been addressed. Your feedback helped us improve the presentation, examine the role of rehearsal-data construction more directly, and assess factual errors after consolidation more carefully.

We appreciate the time you devoted to the paper, your constructive suggestions, and your continued support.

Best regards, The authors

Reviewer_EpvDOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates memory consolidation as recovering context-induced gains after removing the source context while preserving non-target behavior. Cortex generates rehearsal queries and distills a context-conditioned teacher into FFN-only low-rank or structural updates, with a shared expansion for MoEs and optional query replay. The testbed spans five task families and several dense and MoE backbones. The results and controlled ablations provide useful evidence for the integrated design, particularly on 50 sequential factual edits.

Critical strengths
  1. The paper evaluates consolidation together with retention across heterogeneous tasks, and supplies raw component scores rather than only normalized CR/RR. It appropriately distinguishes output suppression from knowledge erasure and finite-state realizability from generalization.

  2. The ablations are substantive: same-data CE/KL controls, FFN versus attention updates, replay, hidden-state alignment, and shared versus independent MoE expansion under both total and active parameter budgets. These experiments support the design choices more convincingly than an aggregate benchmark comparison alone.

  3. The sequential editing result is strong within the evaluated setting: Structural reaches 88.9% CR and 99.8% aggregate RR after 50 edits, with 36.86M accumulated added parameters. The age-aligned curves and raw efficacy, generalization, and portability results help assess acquisition and forgetting separately.

Critical weaknesses
  1. The continual result is narrower than a general persistent-memory claim, and the replay ablation is confounded by optimization budget. All three seeds use the same 50-fact order, and the 100-query buffer can retain the complete generated history. Appendix F.2 also states that all conditions use 30 epochs, so adding up to 50 replay examples to two current examples increases the number of updates. Please include a token/step-matched no-history control and at least one setting that genuinely stresses the replay limit, such as a smaller fixed buffer over the existing sequence, together with alternative edit orders.

  2. The paper reports parameter counts but not the operational cost of consolidation and model-native recall. The method requires query generation, teacher outputs and hidden states, and repeated optimization; some batch settings use 100–200 epochs. Structural capacity grows with each update, and Appendix B.4 notes that folding a shared branch into every MoE expert replicates storage and executes it on each active route.

Questions for authors
  1. How far does the proposed consolidation extend to long-term conversational memory? The agent-experience and factual-portability evaluations are useful, but do not directly test information distributed across multiple sessions, temporal validity, or revisions of previously stored facts. Could you evaluate a representative subset of LoCoMo or LongMemEval by consolidating sessions sequentially and answering held-out questions without the source history or derived textual summaries?

  2. Can you enumerate the tuning/test partition and release the exact judge and baseline settings used for the reported runs? For a small shared set of edits, does the ranking persist with short-answer greedy decoding and a separate factual-correctness score that distinguishes wrong facts from repeated but factually correct answers?

  3. Appendix C appropriately acknowledges SELF-PARAM and DCD, and the paper already compares several context-internalization methods. However, Locas-GLU also introduces a mergeable gated FFN memory branch, while SELF-PARAM already evaluates sequential context injection. Could you clarify the specific algorithmic novelty of Cortex beyond these ingredients, particularly the distinction between Structural and Locas-GLU?

7 comments
AuthorsOpenReview ↗

Response to Reviewer EpvD (Part 1/5, Weakness 1)

Dear Reviewer EpvD,

Thank you for highlighting the value of reporting raw task scores alongside CR and RR and for recognizing the controlled ablations of our design choices. We especially appreciate your observation that tracking each fact from the time of its edit helped distinguish initial learning from later forgetting in the 50-edit sequence.

We have highlighted the revisions in green in the manuscript and respond to your questions and suggestions below.

Response to Weakness1: Edit order and replay

The continual result is narrower than a general persistent-memory claim, and the replay ablation is confounded by optimization budget. All three seeds use the same 50-fact order, and the 100-query buffer can retain the complete generated history. Appendix F.2 also states that all conditions use 30 epochs, so adding up to 50 replay examples to two current examples increases the number of updates. Please include a token/step-matched no-history control and at least one setting that genuinely stresses the replay limit, such as a smaller fixed buffer over the existing sequence, together with alternative edit orders.

Summary: Both Cortex variants had higher mean final CR than all five editing baselines in both orders of the same 50 facts (§5.2, Table 2). On the original order, full replay raised mean final CR over the step-matched no-history control by 86.4 percentage points for Structural and 90.3 for Synaptic (Appendix F.4, Table 10).

1) Edit order. We ran all seven methods on the same 50 ZsRE facts in two orders with Qwen2.5-14B-Instruct. Structural reached 88.9% and 88.6% CR, with aggregate RR near 100% in both orders. Values are means ± sample standard deviations over three runs.

After edit 50 (%)ROMEMEMITAlphaEditWISEFT-MSynapticStructural
Original / CR28.5 ± 1.923.2 ± 2.822.7 ± 3.128.0 ± 3.312.6 ± 0.280.6 ± 3.788.9 ± 3.5
Second / CR29.6 ± 3.126.6 ± 1.824.9 ± 4.231.6 ± 3.111.7 ± 0.483.6 ± 2.488.6 ± 2.6
Original / RR94.4 ± 0.596.4 ± 0.896.0 ± 0.698.9 ± 1.395.0 ± 0.597.2 ± 0.899.8 ± 0.8
Second / RR96.1 ± 0.695.3 ± 0.496.0 ± 1.099.2 ± 1.494.9 ± 0.896.6 ± 1.0100.2 ± 1.0

2) Replay ablations. All four conditions used the original edit order and 30 epochs per edit. Values are means ± sample standard deviations over three runs.

After edit 50Full replaySmall bufferNo replayNo replay, matched optimizer steps
History capacity (questions)1001000
Epochs per edit30303030
Structural CR (%)88.92 ± 3.4566.43 ± 1.3245.94 ± 3.082.48 ± 2.61
Synaptic CR (%)80.55 ± 3.7447.34 ± 4.826.56 ± 1.32−9.72 ± 0.55
Structural RR (%)99.77 ± 0.7999.80 ± 1.86100.15 ± 0.1197.53 ± 1.20
Synaptic RR (%)97.16 ± 0.8195.61 ± 0.6296.13 ± 1.6293.19 ± 1.13
  • Matched optimizer steps. We repeated the current edit’s examples so the no-history control used the same number of optimizer steps as full replay. At that budget, Structural reached 88.92% mean final CR with replay and 2.48% without it. Synaptic reached 80.55% and −9.72%, respectively.
  • Limited buffer. With room for ten historical questions, mean final CR was 66.43% for Structural and 47.34% for Synaptic. This condition also used fewer optimizer steps than full replay, so the comparison does not isolate the effect of buffer capacity.
AuthorsOpenReview ↗

Response to Reviewer EpvD (Part 2/5, Weakness 2)

Response to Weakness2: Consolidation and inference costs

The paper reports parameter counts but not the operational cost of consolidation and model-native recall. The method requires query generation, teacher outputs and hidden states, and repeated optimization; some batch settings use 100–200 epochs. Structural capacity grows with each update, and Appendix B.4 notes that folding a shared branch into every MoE expert replicates storage and executes it on each active route.

Summary: On GPT-OSS-20B, optimization took 1.15 hours for one document and 15.30 hours for 15 documents (Appendix F.7, Table 14). On this model, folding the shared branch into the experts increased storage and reduced mean latency within Transformers. At the largest tested width, the latency increase relative to each unedited SGLang base ranged from 3.12% to 22.66% across the three backbones and two input lengths (Fig. 6).

1) Consolidation. We timed GPT-OSS-20B Structural at width 512. Optimization used two H20 GPUs with data parallelism, batch size four per GPU, and a fixed 200-epoch schedule. All entries are measured wall-clock times (Appendix F.7, Table 14).

StageOne document · 32 QA pairs15 documents · 428 QA pairs
Query generation33 s4 min 24 s
Teacher rollout52 s11 min 29 s
Hidden-state extraction and saving5 s77 s
Parameter optimization1.15 h15.30 h

Training loss often leveled off within the first 50 epochs. The times in Table 14 include all 200 epochs.

2) Inference latency and the cost of folding. On GPT-OSS-20B, the unmerged model evaluates one shared branch per token. Folding stores a copy of its weights in each of the 32 experts and computes the added channels on the four routes active for that token. The GPU can then process those channels within the existing expert computation. The first two rows compare the layouts within Transformers. The last two compare serving backends with the folded layout held fixed.

GPT-OSS-20B, k=1,024Added projection-weight storage (BF16)1,024-token input4,096-token input
Transformers, unmerged0.42 GB93.5 ± 3.2 s105.5 ± 1.5 s
Transformers, merged13.59 GB90.1 ± 2.6 s100.4 ± 2.5 s
SGLang, merged13.59 GB9.2 s10.6 s

The latency columns give the batch means plotted in Fig. 6(c,d), rounded to 0.1 s. Each of the three runs used four prompts, 1,024 generated tokens per prompt, and tensor parallelism 1. Error bars show the SD (below 0.01 s for both SGLang entries). Both tables report storage for the added projection weights in decimal GB, using two bytes per BF16 entry.

Within Transformers, the larger folded layout saved about 3.4 and 5.1 seconds per batch. Serving the same folded weights with SGLang’s expert kernels reduced mean latency by a further factor of about 9.8 and 9.5, respectively.

3) Overhead relative to the unedited SGLang base. We compared each merged model at width 1,024 with its own unedited SGLang base under the same batch and generation settings (Fig. 6). On GPT-OSS-120B, folding stores a copy of the shared weights in each of the 128 experts.

BackboneDistinct added projection weightsAdded weight entries after mergingAdded projection-weight storage (BF16)Latency, 1,024-token inputLatency, 4,096-token input
Qwen2.5-14B (dense)0.755B0.755B1.51 GB+3.40%+3.12%
GPT-OSS-20B (32 experts)0.212B6.79B13.59 GB+22.66%+19.29%
GPT-OSS-120B (128 experts)0.319B40.77B81.54 GB+7.76%+7.67%

At width 1,024, the merged entries add 5.14% to the Qwen2.5-14B base parameter count and 34.9% to the GPT-OSS-120B base count. On GPT-OSS-120B, one BF16 copy of the shared projections occupies 0.64 GB, and the 127 additional copies account for 80.90 GB. The 50-edit experiment used a narrower branch on Qwen2.5-14B. Structural added one channel per edit, accumulating 36.86M weights across the 50 edits, about 0.25% of the base model (Appendix H, Table 21).

AuthorsOpenReview ↗

Response to Reviewer EpvD (Part 3/5, Question 1)

Response to Q1 — Conversational memory and changing facts

How far does the proposed consolidation extend to long-term conversational memory? The agent-experience and factual-portability evaluations are useful, but do not directly test information distributed across multiple sessions, temporal validity, or revisions of previously stored facts. Could you evaluate a representative subset of LoCoMo or LongMemEval by consolidating sessions sequentially and answering held-out questions without the source history or derived textual summaries?

Summary: After 19 and 32 LoCoMo session updates, Structural (k=16) scored 20.99 and 20.19 token F1 on held-out questions without the dialogue at recall. The full-dialogue reference scored 22.57 and 18.87, respectively (Appendix F.5, Table 12). In a separate succession experiment, adding validity years during training and supplying the current date at recall raised final current-president accuracy from 15.3 ± 2.4% to 86.1 ± 20.6%. Accuracy on unchanged capitals stayed at 100% (Appendix F.6).

1) Multi-session recall. The two LoCoMo dialogues [1] span approximately six and eight months. At the final checkpoint, the updated Qwen2.5-14B-Instruct models received only a held-out question and common QA instructions. The full-dialogue reference received the entire conversation. The selected results below come from Table 12. F1 values are means ± sample standard deviations across three training seeds for adapted models and three sampling runs for the references.

Dialogue / metricVanillaFull dialogue in contextLocasSynaptic (r=4)Structural (k=16)
conv-30 · 19 sessions · F1 (%)0.00 ± 0.0022.57 ± 0.222.78 ± 0.0018.72 ± 1.9020.99 ± 1.00
conv-30 · CR (%)0.0100.012.382.993.0
conv-41 · 32 sessions · F1 (%)0.21 ± 0.0018.87 ± 0.071.25 ± 0.3017.34 ± 0.5220.19 ± 0.96
conv-41 · CR (%)0.0100.05.691.8107.1

There were 81 held-out questions on conv-30 and 152 on conv-41. We derived CR from the mean F1 scores in Table 12 using Eq. (3): 100 × (method − vanilla) / (full dialogue − vanilla), separately for each dialogue. On conv-41, Structural's mean F1 was 20.19 versus 18.87 with the full dialogue, so its CR exceeds 100%. (See Appendix F.5 for more details)

2) Answering questions about the current officeholder. When two presidents serve in succession, replaying “Who is the president?” can give the same question two incompatible targets. In the dated condition, the source fact and its rehearsal questions specify the validity year. At evaluation, the user asks the same undated question and the system prompt supplies the current date.

We tested this on four fictional countries that each changed president twice while keeping the same capital. Both conditions used the same update and replay schedules and the same undated questions about the current president and capital. Recall of former presidents for a specified past year remains untested. Appendix F.6 (p. 40) reports the final update. The table gives all three updates for Structural (k=16), as mean ± sample SD over three seeds.

Accuracy (%)President after update 1President after update 2President after update 3Capital at every update
Without dates100.0 ± 0.054.2 ± 7.215.3 ± 2.4100.0 ± 0.0
With validity years and current date100.0 ± 0.070.8 ± 31.586.1 ± 20.6100.0 ± 0.0

In LoCoMo, we also expressed relative dates in generated questions as absolute dates when the session provided enough information (Appendix F.6).

Corrections during replay. In the 50-edit protocol, the frozen pre-update model regenerates answers to historical questions (Appendix F.3, Table 5). Replaying an undated question can reinforce an obsolete answer. For a correction, the new training target is the teacher’s answer given the updated source. If the teacher relies on prior knowledge and repeats the old answer [2], the student receives that answer as its target.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers: Summary of Strengths and Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for reading Cortex closely and for suggesting ways to test how well its results hold across updates.

Strengths highlighted in the reviews

  • A common way to evaluate learning and retention. GYfj noted that the formulation makes concrete “what should be learned from context and what should be preserved in each setting.” EpvD highlighted the raw task scores alongside the normalized consolidation and retention rates, so readers can see the underlying performance on each task.
  • Comparisons across tasks, models, and methods. fqPo highlighted the range of benchmarks, backbones, and baselines, and the trade-off the comparisons reveal between learning from a context and retaining other capabilities. GYfj valued using the same two objectives across tasks as different as factual editing and behavioral steering.
  • Controlled tests of the design choices. EpvD and GYfj singled out the comparisons using the same training data, the FFN and attention ablations, and the tests of replay and shared MoE expansion. fqPo also noted that the ablations help explain the contributions of the update location and training method.
  • Distinguishing initial learning from later forgetting. EpvD described the result as “strong within the evaluated setting” and highlighted the analysis of recall over time. Structural reached 88.9% CR and 99.8% aggregate RR after 50 edits. Tracking accuracy from the point when each fact was introduced helped distinguish a weak initial write from a fact recalled less reliably after later edits.

Manuscript revisions

We have highlighted manuscript revisions in green. Most of the new experiments and protocol details appear in Appendix F, with a guide in Appendix A. The main text reports the second edit order (Table 2) and inference latency (Fig. 6) and discusses the baseline results in §5.1.

Topic (reviewer)Revision and findingWhere to look
Edit order and replay (EpvD)We tested a second order of the same 50 facts and a 10-question buffer. At matched optimizer steps, replaying history gave higher final CR than repeating current examples (88.92% vs. 2.48% for Structural and 80.55% vs. −9.72% for Synaptic).§5.2, Table 2 and Appendix F.4, Table 10
Dialogue recall and capacity (EpvD, GYfj)After 19 and 32 LoCoMo sessions, Structural (k=16) scored 20.99 and 20.19 F1 without the dialogue. The full-dialogue references scored 22.57 and 18.87 (93.0% and 107.1% CR for Structural). Increasing k from 4 to 16 gave no consistent F1 gain.Appendix F.5, Table 12 (LoCoMo) and §5.3, Fig. 5n (50-edit retention)
Changing tasks and facts (EpvD, GYfj)Editing CR held steady after steering on both backbones. Final API CR matched its first-stage value on Qwen2.5-14B and was 3.8 points below its first-stage value on GPT-OSS-20B. In a separate experiment, validity years during training and the current date at recall raised final current-president accuracy from 15.3% to 86.1%. Capital accuracy stayed at 100%.Appendix F.6, Table 13 and the following paragraph
Cost and capacity (EpvD)On GPT-OSS-20B at k=1,024, folding the shared branch into all 32 experts traded extra weight storage for lower mean batch latency compared with running the branch separately in Transformers. At that width, latency overhead over each unedited SGLang base for 1,024/4,096-token inputs was 3.40%/3.12% (Qwen2.5-14B), 22.66%/19.29% (GPT-OSS-20B), and 7.76%/7.67% (GPT-OSS-120B). Across 50 edits, Structural added 36.86M parameters to Qwen2.5-14B (about 0.25%).Fig. 6 (latency), Table 21 (parameters), and Appendix F.7, Table 14 (measured writing times)
Evaluation protocol (EpvD, GYfj)We clarified the tuning and evaluation splits, added configurations for five editing baselines, and scored greedy answers with a rubric that credits repeated correct answers. Both Cortex variants reached 95.33% mean efficacy under this protocol.Appendix F.1 (tuning), Tables 6 and 11 (settings and results), and Appendix G.2.1 (original editing rubric)
Presentation and prior work (fqPo, EpvD, GYfj)We clarified the axes in Fig. 3, discussed the baseline results, related rehearsal questions to prior work, and compared Cortex with Locas-GLU, SELF-PARAM, and DCD.§3.1 and 5.1, Fig. 3 caption, Appendices C and F.2

The response to fqPo compares generic and task-specific rehearsal questions (W2) and reports a SimpleQA error analysis (Q1). On 1,000 unrelated questions, the incorrect-answer rate rose from 72.90% before editing to 93.90% for Structural and 95.73% for Synaptic after 50 edits. The share graded “not attempted” fell from 22.50% to 1.30% and 0.37%, respectively. Accuracy was 4.60%, 4.80%, and 3.90% in the same order.


Best regards,

The authors of Cortex

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.