- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates continual in-context learning as preserving ICL performance as an agent's experience outgrows its window, and identifies two requirements for repeated KV compaction: each module must survive appended context, and modules from successive compactions must compose. The proposed method meets both by distilling self-study from the oldest experiences into a new KV cartridge, initialized from ridge-regularized Attention Matching and trained behind the frozen stack. It reports roughly 13%, 60% and 18% relative gains over the strongest same-model baselines on three benchmarks at 19 to 23× compression.
Critical strengths
-
The problem is well posed, and the failure analysis, separating survival of appended context from composition across modules, is a contribution independent of the proposed method.
-
Evaluation is broad and largely budget-matched, covering recall, reasoning over a stream and agentic tool use against text-based, weight-based and KV baselines.
-
Gains are consistent across these regimes at high compression, and the claims are hedged where the evidence is thin.
Critical weaknesses
-
The distinguishing mechanism, training each module behind the frozen stack, is ablated only on the recall benchmark and worth a few points there, while the large composability effect comes from the synthetic task and the stream and agentic settings ablate only cadence, capacity and representation.
-
On the most realistic benchmark, retrieval appears only as an oracle that still beats the method by 11 points, and the document store is never compacted, so the 12M-token figure reflects what tools reach rather than what the memory covers.
-
Statistically thin evaluation: no intervals are reported for the stream, database or agentic benchmarks, and the database accuracy gap of 77.5% versus 71.7% over 20 questions, from two method runs against three baseline runs, is about one question.
Questions for authors
-
Can you ablate the consolidation mechanism, stack-aware training and the initialization, on the stream and agentic benchmarks rather than only on the synthetic and recall tasks?
-
Can you add a non-oracle retrieval baseline on the agentic benchmark, using the same tools the agent already has, since the oracle still outperforms the memory by 11 points?
-
How many runs underlie the stream and agentic results, and can you report intervals for the 0.358 mean IoU and the 111 graded criteria, which come from a single judge pass?
-
Can you report the database task with the original ordering and the recurrent-state-preserving configuration, so that setting matches the configuration and compression ratio used elsewhere?
5 comments
Reply by Authors (2/2)
Results hold over independent seeds on BSM, Database and Harvey LAB (W3, Q3)
Blind Spectrum Monitoring. The results hold across independent seeds. We reran every cadence on Qwen3.5-9B and Qwen3-32B with three seeds, and each seed regenerates its own trajectory and self-study data. The seed means stay within 0.013 of the submitted numbers (Table 4).
Table 4. BSM IoU, submission vs. mean ± s.d. over 3 independent seeds.
| Cadence | Qwen3.5-9B, submission | Qwen3.5-9B, 3 seeds | Qwen3-32B, submission | Qwen3-32B, 3 seeds |
|---|---|---|---|---|
| Every 15 scans | 0.358 | 0.345 ± 0.017 | 0.345 | 0.345 ± 0.049 |
| Every 30 scans | 0.320 | 0.324 ± 0.012 | 0.276 | 0.270 ± 0.037 |
| Every 45 scans | 0.255 | 0.263 ± 0.011 | 0.246 | 0.246 ± 0.046 |
Database. Over 40 questions and three seeds, Accordion has the highest mean benchmark score and is within noise of the top baselines (Table 5). In the updated results, on accuracy, Accordion leads KVzip by about 1.2 questions, AM only by about 3, ICL by about 6 and ACE by about 11.
Table 5. Database, 40 questions, 3 seeds, original ordering and GDN-state caching.
| Method | Repeats per Day 1 | Mean benchmark score (before shift / after shift) | Mean accuracy (before shift / after shift) | STD across seeds |
|---|---|---|---|---|
| Accordion 20x | 1 | 0.348 (0.371 / 0.326) | 0.533 (0.533 / 0.533) | 0.069 |
| KVzip 20x | 2 | 0.341 (0.364 / 0.318) | 0.504 (0.533 / 0.475) | 0.009 |
| AM only 20x | 2 | 0.317 (0.326 / 0.308) | 0.458 (0.467 / 0.450) | 0.050 |
| ICL | 2 | 0.309 (0.328 / 0.291) | 0.375 (0.425 / 0.325) | 0.034 |
| ACE 20x | 2 | 0.142 (0.203 / 0.081) | 0.271 (0.342 / 0.200) | 0.036 |
Harvey LAB. Following the same setup as the paper, we run Accordion with 3 independent seeds and the tools-only ICL baseline with 7 independent seeds. On average, Accordion (86.7 of 111 criteria) outperforms the seven tools-only ICL runs (77.0) by about 10 criteria, which is smaller than the 14-criterion gap in the single-seed result of the submission. Judge noise is small next to run noise. Two independent Sonnet judges agree on 95.5% of 333 criterion verdicts and change run totals by at most 2 criteria. The revised paper will report these multi-seed means and intervals on all three tasks.
Table 6. Harvey LAB, Qwen3.8-27B. Criteria passed out of 111, higher is better.
| Method | Independent runs | Mean ± s.d. |
|---|---|---|
| ICL (tools only) | 7 | 77.0 ± 5.1 |
| Accordion | 3 | 86.7 ± 5.5 |
Database with the original ordering and GDN-state caching (Q4)
As requested, we rerun Database with the original ordering, GDN-state caching and 20x compression, over all 40 questions and three independent seeds for every method. Each seed samples a fresh Day 1, and all methods continue independently from that same Day 1, which removes noise common to every method. Training-free methods get one extra repeat per seed. Results are in Table 5. Accordion has the highest mean benchmark score at 0.348 and is within noise of ICL and the best compaction baselines. We will replace Database results in the revision with three-seed results.
Interestingly, we find that Accordion is able to handle ground truth distribution shifts even with a frozen stack without retraining or modification. Training each new cartridge in context with older ones allows information to update cleanly. We present Database results with the purpose of matching ICL’s ability to handle problem solving under uncertainty, so Accordion being within noise of ICL and top training-free methods achieves this (under the new setting, this agrees with our submitted result of matching ICL). We strengthen this result by adding that Accordion matches top methods on ground truth shift.
We hope these ablations and three-seed intervals address your reservations about the frozen-stack mechanism and the statistical strength of the evaluation, and we would be grateful if you would consider raising your score.
Thanks for the extensive rebuttal. I appreciate the new results. However, the standard deviations (std) in some of these results seem high (e.g., Accordion 20x and KVzip 20x are close with high std in the Table 5 Database) and only three seeds have been used for these experiments. Therefore, I think my concern about statistically thin evaluation remains so I keep my rating.
Reply by Authors (1/2)
Thank you for the close reading. We appreciate that you saw value in the failure analysis independent of the method, and that you found the evaluation broad and largely budget-matched. Following your suggestions, we ablated the method components on BSM and Database (W1, Q1), ran a non-oracle retrieval baseline on Harvey LAB (W2, Q2), and now report three-seed intervals on BSM, Database and Harvey LAB (W3, Q3). We also reran Database with the original ordering and GDN-state caching (Q4).
Method ablations on BSM and Database (W1, Q1)
Blind Spectrum Monitoring. We ablate each method component one at a time, fixing everything else. Training on top of the frozen stack adds about 0.06 to the BSM score on every seed. Ridge regularization matters within AM initialization, since removing it costs 0.062. All arms in a seed share the same self-study data, so the differences are paired.
Table 1. Component ablations on BSM, Qwen3.5-9B at cadence 15, paired per-seed differences, mean ± s.d. over 3 seeds. Differences are from Accordion trained on the same self-study data (0.327 ± 0.002).
Database. We replay existing Accordion trajectories and change one component at a time, so every arm uses the same trajectory and self-study data. We ablate independent vs. stack-aware training. Independent compaction lowers the score on every seed, by 0.062 on average, with most of that drop coming after the database change at Q21 (learning edits to old knowledge). We will add both ablation tables to the paper.
Table 2. Component ablations on Database, 3 seeds, paired per-seed differences from the control replay (0.347 ± 0.116).
Harvey LAB. Harvey LAB is an agentic task, so its results vary from run to run. Each Harvey LAB ablation arm would need to average results from multiple seeds to show a component's true effect. That cost does not fit in the rebuttal window. The BSM and Database ablations above already isolate the components on shared self-study data. We will add the Harvey component ablations to the revision.
Non-oracle retrieval baselines on Harvey LAB (W2, Q2)
The tools-only baseline in the paper is already a non-oracle retrieval baseline. Its agent searches and reads the same firm document store with the same tools as Accordion. In the paper, Accordion passes 87 of 111 criteria and this baseline passes 73.
As another non-oracle retrieval baseline, we ran retrieval-augmented generation (RAG). Before each task, BM25 uses the task prompt to preload 560K characters of passages from the document base. RAG keeps raw passages in context and compresses nothing, so it sits outside the compression methods we compare against (KVzip, SnapKV, Attention Matching, C²KV and CAS). RAG passes 88 of 111 criteria and Accordion passes 87, which is within run-to-run noise (Table 3). RAG needs far more memory in context to reach that score. We will add this result to the Harvey LAB appendix.
Table 3. Harvey LAB, Qwen3.8-27B, paper setting, single runs. Criteria passed out of 111, higher is better. All arms have the same document tools.