acceptodds
Under review as a conference paper at ICLR 2027

Global Cache Workspace: Towards Robust Reasoning through Internal Computation

Abstract

Language models are usually adapted by updating their weights, trading broad, out-of-domain competence from large pre-training runs for in-domain gains. We show that adaptation can instead live in a model's memory. We introduce the Global Cache Workspace (GCW), a module inspired by Global Workspace Theory that selects information from specialised heads across the layers of a frozen backbone, integrates it in a shared workspace, and broadcasts an update back into their memory, covering both attention and linear-attention layers. Trained with next-token prediction while every backbone parameter stays fixed, GCW reaches in-domain gains competitive with full fine-tuning and LoRA while forgetting far less: across every adaptation setting we study, it loses the least out-of-domain reasoning accuracy on average, and after mathematical adaptation its loss is less than half that of weight-space methods. Its adaptation is also selective. With thinking enabled, GCW outperforms full fine-tuning in domain while shortening in-domain reasoning by up to 82%, a saving comparable to weight-space methods, yet keeps reasoning on unfamiliar tasks close to its original length; weight-space methods shorten that reasoning too, and lose far more accuracy there. Adapting what a frozen model remembers, rather than its weights, thus adds capability while preserving far more of what the model already knows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, Test Time Compute, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GDzwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies adaptation of language models while retaining their previous abilities. It proposes Global Cache Workspace (GCW), which freezes the backbone and periodically compresses per-head KV caches and linear-attention states into workspace vectors. A small set of source heads is selected, their information is integrated by a shared processor, and context-dependent updates are broadcast back to the latest cache positions. Experiments on Llama 3.2 and Qwen3.5 backbones compare mathematical, broad, and sequential adaptation with weight-based and cache-based methods. The paper reports competitive adaptation and less average KOR-Bench forgetting, and uses processor-capacity and source-head ablations to study the workspace design.

Critical strengths
  1. The paper addresses an important problem: models often need to learn new tasks while retaining their existing knowledge and reasoning abilities. Reducing this loss would be useful in many adaptation scenarios, making the research question consequential.
Critical weaknesses
  1. The broad retention claim exceeds the evidence. Lower average forgetting on KOR-Bench does not establish preservation of factual knowledge, language understanding, coding, or mathematical abilities generally. After direct Reasoning Gym adaptation, Qwen3.5-2B's average mathematical accuracy falls from 50.83% to 26.81%, a loss of 24.02 percentage points. These results support lower average forgetting on the reported KOR-Bench evaluations, while substantial losses remain possible on other abilities.

  2. The conclusion that structure matters more than capacity is consistent with the experiments but is not causally established. Increasing processor capacity from 1.5% to 10% of backbone parameters gives no consistent gains under a fixed 2,000-step budget, without evidence that each processor size was sufficiently optimized. Using all heads as sources instead of approximately 10% also brings only small gains, but selecting even one source head still broadcasts to all heads. These ablations therefore do not isolate cross-head communication or the individual architectural components. The fixed window size P = 64 and number of retained older cache positions k = 32 also leave uncertainty about sensitivity to these settings and their effect on accuracy and computational cost.

Questions for authors
  1. KOR-Bench mainly evaluates in-context abstract reasoning with rule descriptions. How should the broad retention claim in lines 349-358 be interpreted given the decline in Qwen3.5-2B's average mathematical accuracy from 50.83% to 26.81%, a loss of 24.02 percentage points, after direct Reasoning Gym adaptation? Could you narrow the claim to lower average forgetting on the reported KOR-Bench evaluations and provide results for individual models and tasks? A broader claim would need support from a more diverse set of out-of-domain evaluations.

  2. The comparison between processors with 1.5% and 10% of the backbone's parameter count in Figure 6b supports the conclusion that additional capacity is not the main bottleneck within the current 2,000-step budget. Would the result change if the learning rate and training duration were tuned separately for each processor size? Could you provide training curves, validation losses, and results at the best checkpoint for each size to distinguish a lack of benefit from additional capacity from insufficient optimization of the larger processor? This distinction matters for the interpretation in lines 134-137.

  3. Since a single selected source head still broadcasts to all heads, which result in lines 419-428 isolates the benefit of cross-head communication? Could you vary head selection, the shared processor, broadcast writeback, positional encoding, and writeback position one at a time, while matching parameter counts and sufficiently tuning hyperparameters, to establish their individual contributions? Until then, could "structure matters more than capacity" be presented as an explanation consistent with the tested models, tasks, and training budget rather than an established causal conclusion?

  4. The cache settings in lines 208-215 use a window of P = 64 tokens and retain k = 32 older cache positions. Since P determines rewriting frequency and k controls the accessible history, could you report a grid ablation over P, k, and input context length? Please include adaptation accuracy, KOR-Bench retention, end-to-end latency, peak GPU memory usage, and computation per generated token to show whether GCW depends on these settings and how they affect adaptation, forgetting, and practical cost.

2 comments

We sincerely thank reviewer GDzw for the careful assessment and the specific suggestions for strengthening the claims and experimental controls. We agree that the measured findings need to be distinguished from their possible explanations.

1. Scope of the retention claim.

The reviewer is correct that lower average KOR-Bench forgetting does not establish general preservation of pretrained capabilities. The Qwen3.5-2B mathematical decline after direct Reasoning Gym adaptation is substantial and is not negated by the KOR-Bench result.

We have qualified both the main and closing retention statements in the Abstract with “as measured by KOR-Bench.” The Discussion now explicitly states that this provides evidence on out-of-domain reasoning tasks, rather than preservation of pretrained capabilities more generally, and expands the acknowledgement of mathematical regressions. Individual benchmark and KOR-Bench category scores remain available in Appendix C, now extended with the 27B broad/sequential results.

2. Capacity and optimisation.

We can confirm that such checks were performed during our experiments: all training and validation curves were checked for convergence, and a subset of settings was evaluated across various training lengths and learning rates to validate our choices. We have added such results concerning training length in the processor size ablation in Appendix C.5 (Processor capacity across training checkpoints), cross-referenced from Section 4.2. The results are summarised in the tables below, which report mean accuracy (%) across five mathematical benchmarks, averaged over three seeds, with the standard deviation across seeds:

Llama 3.2-3B

Processor size1,000 steps2,000 steps3,000 steps
1.5% (default)40.04 ± 0.1240.25 ± 0.0840.81 ± 0.25
5%39.97 ± 0.3440.44 ± 0.3240.21 ± 0.39
10%39.79 ± 0.2740.63 ± 0.3140.61 ± 0.08

Qwen3.5-2B

Processor size1,000 steps2,000 steps3,000 steps
1.5% (default)32.69 ± 0.3132.36 ± 0.2832.14 ± 0.10
5%32.31 ± 0.0732.42 ± 0.0232.39 ± 0.06
10%32.13 ± 0.2832.20 ± 0.1532.39 ± 0.38

The qualitative capacity result persists when training is shortened to 1,000 steps or extended to 3,000 steps: accuracy remains similar across processor sizes, without a consistent advantage for larger processors. At any fixed checkpoint, the spread across sizes is at most 0.60 percentage points for Llama 3.2-3B and 0.56 for Qwen3.5-2B. Thus, the result is not specific to the selected 2,000-step training budget. Taken together, these results support 2,000 steps as a practical common checkpoint across these tests, since extending training to 3,000 steps adds computational cost without a consistent accuracy improvement across processor sizes and backbones.

This directly addresses sensitivity to training length over the tested range; robustness to other optimisation settings for this particular ablation, including learning rate, remains to be tested, but we would be happy to include it in the camera-ready version, should the reviewer find that relevant and necessary.

3. Single-source broadcast and component attribution.

The cited paragraph presents evidence supporting a capacity-limited workspace, i.e., a limit on the number of heads contributing information to update other heads, rather than cross-head communication in general. For cross-head communication, the comparison with BGT is more pertinent: it retains a shared processor across layers but uses sequence positions, rather than individual heads, as workspace tokens (Section 3, paragraph beginning “For BT, the same update”). GCW achieves substantially higher average KOR-Bench retention than BGT across all three adaptation settings, supporting the value of this head-indexed organisation beyond simply sharing a processor across layers (Section 4.1, paragraph “Heads as specialists”; Figure 5).

We are happy to consider additional ablations for the camera-ready version and would welcome clarification on the intended interventions. We will vary the writeback schedule through the window-size ablation requested in Question 4. For "broadcast writeback", does the reviewer mean varying which heads receive updates or how those updates are applied? For "positional encoding", does the reviewer mean the encoding within the cache processor or the handling of positional information from the frozen backbone? Our initial architectural exploration included alternative locations for computing selection scores, learned selection scores, and different processor depths and head counts. These exploratory comparisons produced small differences and were omitted to keep the paper focused, but we would be happy to include those most relevant to the reviewer’s concerns.

4. Sensitivity to P, k, context length, and practical cost.

We thank the reviewer for this pertinent suggestion. We will include this ablation in the camera-ready version.

Reviewer_ZaYpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Global Cache Workspace, a novel framework for adapting frozen LLMs to new tasks while mitigating catastrophic forgetting. GCW introduces a trainable module that dynamically reads, integrates, and modifies the model's KV cache. Experiments across different model sizes and adaptation settings demonstrate that GCW achieves competitive in-domain performance compared with full fine-tuning and LoRA, while substantially reducing out-of-domain performance degradation in many settings.

Critical strengths
  1. Novel and potentially impactful research direction. I consider this a pioneering work exploring a promising alternative paradigm for LLM adaptation. The central hypothesis that adapting a model's internal memory can enable task adaptation while mitigating catastrophic forgetting is meaningful. This direction could potentially open new opportunities for parameter-efficient and continual adaptation of LLMs.

  2. Well-motivated and clearly presented methodology. The paper is well written and well motivated. The proposed architecture provides a concrete approach to the central hypothesis through selective head-level information integration and cache updates.

  3. Promising empirical results. The experimental results demonstrate that GCW can achieve competitive in-domain performance while preserving substantially more out-of-domain capabilities than conventional weight-based fine-tuning methods in several settings. The results suggest that memory-based adaptation is a promising direction worth further investigation.

Critical weaknesses
  1. Unclear scalability to larger LLMs. My major concern is that the relative advantage of GCW over conventional fine-tuning methods appears to diminish as the backbone model size increases. In the mathematical adaptation experiments, GCW substantially outperforms full fine-tuning and LoRA in out-of-domain retention on smaller Qwen3.5 models, but this advantage becomes much smaller at 27B and even disappears relative to full fine-tuning under Thinking-enabled evaluation. However, the experiments stop at 27B, and the broad and sequential adaptation experiments cover even smaller models. It remains unclear whether the proposed method can maintain its advantages over LoRA and full fine-tuning when applied to larger LLMs.

  2. Insufficient analysis of task-dependent head selection. A central motivation of the proposed framework is that context-dependent cache adaptation can help preserve out-of-domain capabilities. However, the paper does not provide sufficient analysis of how source-head selection varies across different tasks. In particular, the GCW processor is trained on in-domain data, while source heads are dynamically selected according to a cache-magnitude-based heuristic. It remains unclear whether different task domains actually lead to substantially different selected heads, or whether the same high-salience heads are consistently selected across tasks. An analysis of head-selection frequencies, cross-domain overlap, and cache-update magnitudes would help explain how the proposed mechanism achieves selective adaptation and reduced catastrophic forgetting.

  3. Unclear adaptation–retention trade-off relative to conventional fine-tuning. Although GCW demonstrates better out-of-domain retention in many settings, its in-domain performance is not consistently superior to LoRA or full fine-tuning. In some settings, the proposed method provides limited adaptation gains or even reduces in-domain performance relative to the original backbone. It is therefore unclear whether GCW fundamentally improves the trade-off between acquiring new capabilities and preserving existing ones, or whether part of its retention advantage reflects a more conservative adaptation mechanism. A more systematic comparison of the in-domain performance versus out-of-domain retention Pareto frontier would help establish its advantages.

Questions for authors

See weaknesses.

1 comment

We sincerely thank reviewer ZaYp for the encouraging assessment of memory-based adaptation and the constructive questions about scaling, selection, and the adaptation–retention trade-off.

1. Scaling and the additional 27B results.

We have expanded the reported broad and sequential adaptation comparisons to include Qwen3.5-27B. These results are now included across all relevant sections of the paper. KOR-Bench accuracies at 27B are:

Adaptation settingGCWFull fine-tuningLoRA
Direct Reasoning Gym64.5664.8859.28
Sequential61.7656.4055.92

GCW exceeds both weight-space baselines after sequential adaptation; after direct Reasoning Gym adaptation, it exceeds LoRA and scores 0.32 percentage points below full fine-tuning. Together with the mathematical-adaptation exception you identify, these results limit any claim of a uniformly favourable scaling trend.

The Discussion also now notes that stronger baseline retention appears to contribute to the narrowing gap at higher scales. Additionally, while we are unable to train and evaluate models larger than 27B with our computational resources, we will look to include more models in the intermediate range (4B to 27B) for the camera-ready version to assess whether this narrowing gap is linked to model size.

2. Task-dependent source selection.

We thank the reviewer for this pertinent suggestion. We will analyse source-head selection frequencies, cross-domain overlap in selected heads, and cache-update magnitudes to assess how selection and updates vary across tasks, and include these analyses in the camera-ready version.

3. Improved trade-off versus conservative adaptation.

We agree that retention should be assessed jointly with adaptation. The Results explicitly note that full fine-tuning and LoRA achieve the largest average in-domain gains (Section 4.1, lines 385–387), while the Discussion acknowledges that GCW does not improve in-domain accuracy on the Llama 3.2 backbones or on Qwen3.5-2B with thinking disabled (Section 5, lines 497–500). Our claim is therefore a favourable adaptation–retention trade-off in the evaluated settings, rather than uniformly superior adaptation.

Figure 2 already plots in-domain accuracy change against KOR-Bench accuracy change relative to the matched backbone, showing individual backbones and thinking modes alongside method means. The Results also identify nearly matched adaptation outcomes (Section 4.1, lines 390–392): with thinking disabled, GCW and full fine-tuning achieve mathematical accuracies of 54.74% and 54.73% on Qwen3.5-4B, with KOR-Bench accuracies of 47.76% and 26.64%; GCW and LoRA achieve 62.98% and 62.97% on Qwen3.5-27B, with KOR-Bench accuracies of 49.84% and 40.24% (Tables 3 and 4). These comparisons show that higher retention can accompany comparable adaptation, although they do not rule out more conservative adaptation in other settings.

Figure 2 directly presents the requested adaptation–retention comparison, including operating points with nearly identical adaptation and substantially different retention. We would welcome clarification if the reviewer has a particular additional comparison or presentation in mind, and will be happy to include it in the camera-ready version.

Reviewer_AowaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of adapting to new distributions without degrading the base model's capability to generalize to other domains. Inspired by global workspace theory, the paper proposes Global Cache Workspace (GCW), which adapts models by modifying their KV cache at the latest token position. Concretely, GCW divides contexts into fixed-size chunks, e.g., 64 tokens, and trains an additional processor to mix information from different tokens and different attention heads across different layers, thereby constructing a global workspace. The experiments train five models of different sizes on math reasoning tasks and broader reasoning tasks, and evaluate performance on reasoning benchmarks and KOR-Bench. The results suggest that GCW can achieve a good balance between ID and OOD performance relative to the evaluated baselines.

Critical strengths
  1. Learning to adapt to new domains without hurting previously learned knowledge and skills is an important and timely topic.

  2. The approach is very interesting, since it updates the KV cache at only a single token position per chunk in full-attention layers. It also keeps the backbone parameters frozen and trains a relatively small processor, making it parameter-efficient.

Critical weaknesses
  1. I find the paper in general quite hard to follow. For example, I am not sure about the connection between the problem and the gap. The gap seems to be mostly about communication between different attention heads, but the problem the paper is targeting is forgetting during adaptation. Also, the method section is very dense and notation-heavy. The figures provided do not offer enough detail about the approach. I highly recommend writing down the steps in detail, using an example, and specifying the inputs and outputs of each step.

  2. The benefits of training with GCW are unclear from the current results. The ID improvements are quite mixed; for example, on matched mathematical benchmarks, GCW does not consistently outperform the backbone baseline, while its performance on KOR-Bench is worse. This raises the question of what is actually learned by GCW, especially considering that GCW also lowers the truncation rate on the Qwen backbones. Also, GCW modifies fewer KV-cache positions than BT and BGT, since it only updates the latest position of each head, which could be one reason why it forgets less on average.

  3. The contributions of the different components are unclear. For example, the ablations show that a single source head on Llama 3.2-1B achieves performance comparable to many more source heads, and that a smaller processor achieves performance comparable to processors with many more parameters. This raises the question of how much of the improvement comes from head communication, and how much comes from pooling information from relevant past tokens and writing it into the latest KV-cache entry, potentially improving information retrieval. The comparisons to the baselines are not fully controlled either, as BT and BGT do not read or modify the states of linear-attention layers, whereas GCW does.

Questions for authors
  1. How does GCW relate to test-time training approaches? GCW differs from approaches like LoRA, which change the effective weights used for prediction. Instead, it modifies runtime memory, including the KV cache at the latest position, using the previous context. Therefore, I think it would be interesting to discuss the connections to TTT methods and extend the experiments to much longer contexts to see how GCW behaves with more historical information available for selection and pooling.

  2. What would the performance of the different models be with substantially larger generation budgets? The generation budgets used so far are quite small—I wonder whether truncation is a major factor in the reported performance, especially given the relatively small ID differences.

  3. The BT baseline does not use the same configuration and training protocol as the original paper. For example, the original paper used a larger processor and a backbone fine-tuning stage before training the processor. How does performance compare when using the original BT configuration and training protocol?

  4. It would be interesting to know what the processor produces, i.e., what information is added to the KV cache. This could also help us better understand what GCW learns.

3 comments

We sincerely thank reviewer Aowa for the detailed assessment and for recognising the importance of the adaptation problem and the potential of a small processor acting on a frozen model’s memory. Your comments helped us identify several places where the explanation and interpretation needed to be clearer.

Weaknesses

1. Motivation and step-by-step explanation.

Our motivating question is whether context-conditioned memory updates offer a different adaptation–retention trade-off from modifying backbone weights. Head-level communication is our proposed architecture for computing those updates; we do not claim that insufficient communication causes forgetting and do not investigate this particular question in this work.

We have substantially revised Figure 1 and Section 3. The figure now separates the frozen backbone from the trainable encoder, Transformer, decoder, and gate, and numbers the four stages from cache encoding to gated writeback. The Methods text now explains the processing window, recent/older-position selection, pooling, and gate explicitly; defines the total head count; clarifies TopK and the separate hybrid source budgets; and distinguishes all-head queries from selected-source keys and values. We also explain BT/BGT’s position-indexed workspace and which entries their update rewrites.

The Introduction adds a two-layer, four-head example with one selected source. It illustrates how a head’s representation can become available to all four heads, including unselected heads.

2. Mixed adaptation gains and truncation.

The reviewer is correct that GCW does not consistently improve mathematical accuracy over the backbone and can also reduce KOR-Bench accuracy. Our positive retention result concerns a smaller average loss relative to adaptation baselines, not an absence of forgetting. We have made the mathematical regressions more explicit in the Discussion and qualified the Abstract’s retention claim by KOR-Bench.

There are nevertheless settings with comparable adaptation and different retention. After mathematical adaptation on Qwen3.5-4B with thinking disabled, GCW and full fine-tuning achieve mathematical averages of 54.74% and 54.73%, while their KOR-Bench accuracies are 47.76% and 26.64% (Appendix C.1). This supports a useful trade-off at that operating point, not universal superiority.

We also agree that truncation matters. After mathematical adaptation, under the 4,096-token thinking-enabled budget, the backbones reach the cap on 39%–84% of in-domain responses. Reduced truncation can therefore contribute materially to accuracy gains. The Discussion acknowledges this limitation. Our exploratory backbone evaluations at larger budgets show little change in accuracy (see our response to Question 2 below), although we do not have a comprehensive larger-budget comparison of the adaptation methods and cannot rule out changes in their relative ordering. Reducing the number of output tokens required for a correct answer is itself an improvement; lower truncation rates with maintained or improved accuracy are therefore a positive result.

3. Component contributions and baseline fairness.

A single source does not remove cross-head communication. All heads still query that source and receive updates, as we now explain explicitly in Section 4.2. The result shows that a narrow contribution bottleneck can suffice in the tested setting, not that competition or communication has been causally isolated. Similarly, the capacity experiment establishes no consistent gain from larger processors under the evaluated budget, rather than ruling out gains after size-specific optimisation.

Pooling, fewer rewritten positions, and access to linear-attention states remain plausible contributors. We expanded the hybrid-memory explanation in Section 3. BT/BGT address individual sequence positions, whereas linear attention stores compressed history in a recurrent matrix without separately addressable past positions. Extending that interface requires an additional architectural adaptation. This explains the implementation choice but does not eliminate the confound. The revised Discussion now explicitly acknowledges these architectural differences, which we will investigate through ablations for the camera-ready version.

Questions (1/2)

1. Test-time training and longer contexts. Thank you for this suggestion. Section 2 now discusses TTT more explicitly, including classical TTT (Sun et al., 2020), TTT layers (Sun et al., 2024), and TTT-E2E (Tandon et al., 2025). These approaches use self-supervised learning at inference to update model parameters or a model-valued recurrent state. GCW instead computes memory updates through an offline-trained processor; neither the backbone nor processor parameters change during inference, and no inference-time gradient descent is performed.

Both approaches use context to modify the state supporting subsequent predictions. The distinction is how these updates are learned: TTT performs online optimisation, whereas GCW applies an offline-trained processor to the current context. Longer-context experiments would help assess how GCW uses additional historical information for selection and pooling, and we will include these experiments in the camera-ready version.

2. Generation budgets. We agree with the reviewer that truncation is a meaningful signal to monitor, which is why we have included response length and truncation metrics in our detailed results. Our generation budgets are 2,048 and 4,096 tokens with thinking disabled and enabled, respectively. Released reference implementations use the following limits:

These are implementation-specific precedents, not universal benchmark standards or evidence that the budgets suffice for thinking-enabled models.

That said, we did consider the same question and ran a small number of experiments with 512, 1K, 2K, 4K, 8K, and 32K generation token limits in order to confirm our results wouldn't change drastically under our final settings. If the reviewer still believes this to be a useful addition, we would be happy to extend it and include it in the camera-ready version. In the meantime, we include a small ablation for the Qwen3.5-2B backbone with thinking enabled, to illustrate our results. The following exploratory evaluations compare larger generation budgets. Accuracy is the unweighted mean over GSM8K, SVAMP, and GSM8K-Hard.

Generation budget (tokens)Mean accuracy (%)Reported truncation (%)
4,09634.3780.94
8,19233.9781.01
32,76833.3881.01

These runs show no accuracy improvement with larger budgets for this backbone. Evaluation batch size differed between the 4K and larger-budget runs (32 versus 64), which may explain the slight difference in truncation rates and accuracies, but otherwise, and looking at specific answer samples, it is clear that the answers which get truncated often entered a "doom-looping" behaviour where the model never finds an answer but keeps generating.

Questions (2/2)

3. Original BT configuration and training protocol.

The main BT/GCW comparison uses matched processor budgets and a common frozen-backbone setting for fair comparison, rather than reproducing every aspect of the original BT protocol. Appendix C.4 now includes a supplementary comparison with GCW using the original paper's settings. The five-benchmark mean accuracies are:

BackboneFull fine-tuningBTGCW, 25% sources
Llama 3.2-1B27.1628.2828.98
Llama 3.2-3B38.8640.7041.16
Qwen3-0.6B39.3239.6840.14

GCW has the higher mean in these three reported configurations, although not on every benchmark. The appendix provides individual scores. This adds task-accuracy evidence, not a retention comparison or a significance test. The BT paper uses default generation token limits and a matched learning rate for comparison with FFT; these settings yield lower overall scores. Protocol equivalence requires documentation of processor sizes, backbone fine-tuning, processor-training budget, invocation schedule, and evaluation settings.

4. What the processor writes.

The method specifies the outputs computationally: gated residual KV updates and, for linear-attention heads, rank-one state updates, but we have not identified their semantic content. Training uses the downstream next-token objective rather than cache reconstruction. We therefore cannot infer that an update encodes a particular fact or intermediate calculation from its form alone. Mechanistic analysis of the changes in representations would be valuable, but we consider it beyond the scope of the present study and leave it to future work.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank you for your careful feedback and constructive suggestions. We apologise for the delayed response due to illness within the team. We have revised the manuscript to improve the presentation, expand the reported comparisons, and clarify the scope of the evidence. The main changes relative to the submitted version are:

  • Clearer method presentation. We redesigned Figure 1 with four numbered processing stages and distinct frozen/trainable regions, expanded the definitions and explanations in Section 3, and added an illustrative two-layer example in the Introduction.
  • Additional reported results. Appendix C now includes Qwen3.5-27B results for broad and sequential adaptation, with corresponding updates to Figure 5 and Figures 9–12. Appendix C.4 adds a supplementary BT comparison with GCW using 25% of heads as sources.
  • More precise interpretation. The Abstract and Discussion explicitly qualify the retention evidence by KOR-Bench. We expanded the discussion of mathematical regressions, scaling, and architectural differences that prevent causal attribution to head-level communication alone.
  • Expanded related work. A dedicated paragraph distinguishes GCW from test-time training methods.

We address your individual concerns below, distinguishing the available evidence from questions that still require additional experiments.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.