- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies adaptation of language models while retaining their previous abilities. It proposes Global Cache Workspace (GCW), which freezes the backbone and periodically compresses per-head KV caches and linear-attention states into workspace vectors. A small set of source heads is selected, their information is integrated by a shared processor, and context-dependent updates are broadcast back to the latest cache positions. Experiments on Llama 3.2 and Qwen3.5 backbones compare mathematical, broad, and sequential adaptation with weight-based and cache-based methods. The paper reports competitive adaptation and less average KOR-Bench forgetting, and uses processor-capacity and source-head ablations to study the workspace design.
Critical strengths
- The paper addresses an important problem: models often need to learn new tasks while retaining their existing knowledge and reasoning abilities. Reducing this loss would be useful in many adaptation scenarios, making the research question consequential.
Critical weaknesses
-
The broad retention claim exceeds the evidence. Lower average forgetting on KOR-Bench does not establish preservation of factual knowledge, language understanding, coding, or mathematical abilities generally. After direct Reasoning Gym adaptation, Qwen3.5-2B's average mathematical accuracy falls from 50.83% to 26.81%, a loss of 24.02 percentage points. These results support lower average forgetting on the reported KOR-Bench evaluations, while substantial losses remain possible on other abilities.
-
The conclusion that structure matters more than capacity is consistent with the experiments but is not causally established. Increasing processor capacity from 1.5% to 10% of backbone parameters gives no consistent gains under a fixed 2,000-step budget, without evidence that each processor size was sufficiently optimized. Using all heads as sources instead of approximately 10% also brings only small gains, but selecting even one source head still broadcasts to all heads. These ablations therefore do not isolate cross-head communication or the individual architectural components. The fixed window size P = 64 and number of retained older cache positions k = 32 also leave uncertainty about sensitivity to these settings and their effect on accuracy and computational cost.
Questions for authors
-
KOR-Bench mainly evaluates in-context abstract reasoning with rule descriptions. How should the broad retention claim in lines 349-358 be interpreted given the decline in Qwen3.5-2B's average mathematical accuracy from 50.83% to 26.81%, a loss of 24.02 percentage points, after direct Reasoning Gym adaptation? Could you narrow the claim to lower average forgetting on the reported KOR-Bench evaluations and provide results for individual models and tasks? A broader claim would need support from a more diverse set of out-of-domain evaluations.
-
The comparison between processors with 1.5% and 10% of the backbone's parameter count in Figure 6b supports the conclusion that additional capacity is not the main bottleneck within the current 2,000-step budget. Would the result change if the learning rate and training duration were tuned separately for each processor size? Could you provide training curves, validation losses, and results at the best checkpoint for each size to distinguish a lack of benefit from additional capacity from insufficient optimization of the larger processor? This distinction matters for the interpretation in lines 134-137.
-
Since a single selected source head still broadcasts to all heads, which result in lines 419-428 isolates the benefit of cross-head communication? Could you vary head selection, the shared processor, broadcast writeback, positional encoding, and writeback position one at a time, while matching parameter counts and sufficiently tuning hyperparameters, to establish their individual contributions? Until then, could "structure matters more than capacity" be presented as an explanation consistent with the tested models, tasks, and training budget rather than an established causal conclusion?
-
The cache settings in lines 208-215 use a window of P = 64 tokens and retain k = 32 older cache positions. Since P determines rewriting frequency and k controls the accessible history, could you report a grid ablation over P, k, and input context length? Please include adaptation accuracy, KOR-Bench retention, end-to-end latency, peak GPU memory usage, and computation per generated token to show whether GCW depends on these settings and how they affect adaptation, forgetting, and practical cost.
2 comments
3. Single-source broadcast and component attribution.
The cited paragraph presents evidence supporting a capacity-limited workspace, i.e., a limit on the number of heads contributing information to update other heads, rather than cross-head communication in general. For cross-head communication, the comparison with BGT is more pertinent: it retains a shared processor across layers but uses sequence positions, rather than individual heads, as workspace tokens (Section 3, paragraph beginning “For BT, the same update”). GCW achieves substantially higher average KOR-Bench retention than BGT across all three adaptation settings, supporting the value of this head-indexed organisation beyond simply sharing a processor across layers (Section 4.1, paragraph “Heads as specialists”; Figure 5).
We are happy to consider additional ablations for the camera-ready version and would welcome clarification on the intended interventions. We will vary the writeback schedule through the window-size ablation requested in Question 4. For "broadcast writeback", does the reviewer mean varying which heads receive updates or how those updates are applied? For "positional encoding", does the reviewer mean the encoding within the cache processor or the handling of positional information from the frozen backbone? Our initial architectural exploration included alternative locations for computing selection scores, learned selection scores, and different processor depths and head counts. These exploratory comparisons produced small differences and were omitted to keep the paper focused, but we would be happy to include those most relevant to the reviewer’s concerns.
4. Sensitivity to P, k, context length, and practical cost.
We thank the reviewer for this pertinent suggestion. We will include this ablation in the camera-ready version.
We sincerely thank reviewer GDzw for the careful assessment and the specific suggestions for strengthening the claims and experimental controls. We agree that the measured findings need to be distinguished from their possible explanations.
1. Scope of the retention claim.
The reviewer is correct that lower average KOR-Bench forgetting does not establish general preservation of pretrained capabilities. The Qwen3.5-2B mathematical decline after direct Reasoning Gym adaptation is substantial and is not negated by the KOR-Bench result.
We have qualified both the main and closing retention statements in the Abstract with “as measured by KOR-Bench.” The Discussion now explicitly states that this provides evidence on out-of-domain reasoning tasks, rather than preservation of pretrained capabilities more generally, and expands the acknowledgement of mathematical regressions. Individual benchmark and KOR-Bench category scores remain available in Appendix C, now extended with the 27B broad/sequential results.
2. Capacity and optimisation.
We can confirm that such checks were performed during our experiments: all training and validation curves were checked for convergence, and a subset of settings was evaluated across various training lengths and learning rates to validate our choices. We have added such results concerning training length in the processor size ablation in Appendix C.5 (Processor capacity across training checkpoints), cross-referenced from Section 4.2. The results are summarised in the tables below, which report mean accuracy (%) across five mathematical benchmarks, averaged over three seeds, with the standard deviation across seeds:
Llama 3.2-3B
Qwen3.5-2B
The qualitative capacity result persists when training is shortened to 1,000 steps or extended to 3,000 steps: accuracy remains similar across processor sizes, without a consistent advantage for larger processors. At any fixed checkpoint, the spread across sizes is at most 0.60 percentage points for Llama 3.2-3B and 0.56 for Qwen3.5-2B. Thus, the result is not specific to the selected 2,000-step training budget. Taken together, these results support 2,000 steps as a practical common checkpoint across these tests, since extending training to 3,000 steps adds computational cost without a consistent accuracy improvement across processor sizes and backbones.
This directly addresses sensitivity to training length over the tested range; robustness to other optimisation settings for this particular ablation, including learning rate, remains to be tested, but we would be happy to include it in the camera-ready version, should the reviewer find that relevant and necessary.