- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formalizes test-time continual learning in the observation-driven regime and proposes C3, which trains a bounded-state consolidation operator offline by context distillation from a frozen full-context teacher, so evicted blocks are folded into fixed-size KV-row state in one forward pass at deployment. It reports that C3 beats gradient-based sleep-time consolidation at far lower deployment compute and beats matched-byte memory baselines, with the learned state occasionally exceeding full-history ICL.
Critical strengths
-
Clear formulation of test-time continual learning as constraints plus capabilities, with evaluations mapped to each requirement.
-
Deployment compute is measured for the main comparisons, including supervision generation, and the oracle-GD control is informative.
-
Diagnostics separate acquisition, retention and interference rather than only ranking methods.
Critical weaknesses
-
The central accuracy claim is weak: C3 is trained to reproduce full-history ICL, but it does not exceed that teacher on any task except TREC, where the margin is 0.01 to 0.03 and no intervals or seeds are reported.
-
The stated distinctions from the closest prior work are the consolidation rule (versus CCM) and the continual-learning objective (versus Still), yet neither method is evaluated.
-
The organizing claim, that learning the update rule offline beats optimizing state during deployment, is tested only at 1.7B on 8K to 16K streams, and the one replay-based gradient variant is given storage outside the stated budget.
Questions for authors
-
Can you evaluate CCM and Still under the same streaming protocol and byte budget, since your related work distinguishes C3 from CCM by the consolidation rule and from Still by the continual-learning objective, yet neither appears in the experiments?
-
Can you add intervals and multiple seeds to Figures 3, 4, 11 and 13, and does the TREC advantage over full-history ICL survive?
-
Which evaluation tasks appear in the decay mixture, and can you report format-held-out banking77 and NIAH variants?
-
Can GD be given a retention mechanism that respects the byte budget, and can the asymmetric YaRN factor at 131K in EventQA be justified?
4 comments
Thanks for the rebuttal and the new results. While the revised C3 looks better than the submitted version, some concerns remain. In the 128K NIAH multikey panel, both merge rules sit at the no-memory level for evidence more than 32K tokens back, so retention beyond the native window is not yet demonstrated. The revised rule is also compared only against the teacher and the previous rule, not against the memory baselines, and the ablations remain a single training seed. For these reasons, I keep my negative rating.
We want to thank the Reviewer for their fast reply and for looking closely at the new results. We have updated the PDF so that every figure uses the revised C3, including the comparisons with all memory baselines, which is uploaded in the following anonymized link for convenience: https://anonymous.4open.science/r/ICLR-C3-5573/ctx8b_r2048_grid.pdf
In Figure 18 we show the performance versus evidence age at 32k and 64k context length, where C3 demonstrates retention beyond the native 32k context window.
We agree that at 128K, NIAH retention beyond 32K is weak. For needles more than 32K tokens back, C3 averages .05 (at most .13) against 0 for no memory. However, in this setting the same 2048 rows hold all 63 evicted blocks (126k tokens), where there is much more competition for admission than within a 32k token stream. On TriviaQA at 128K, C3 stays above no memory at every distance (.32 against .21 for evidence more than 32K back.
We could not train additional seeds within the discussion period. The ablation conclusions we draw rest on differences of .10 or more (Appendix D), well above the .03 we treat as inconclusive.
We hope these clarifications address the Reviewer’s remaining concerns, and we thank them again for their time and careful reading.
Thanks for the reply, updated PDF and clarifications. These are nice but I think that the real value of the context compression would lie in the case of longer context (e.g., at or beyond 128K), for which the current results seem inconclusive. Therefore, I thank the authors for the effort but keep my evaluation as is.
Thank you for recognizing the formulation, compute accounting, and acquisition and retention diagnostics. Your questions prompted us to clarify how we compare with the full-history teacher and exactly which gradient-based comparisons fit the memory budget.
W1. Comparison with the full-history teacher. We agree that the revised C3 falls below full-history ICL on TREC, and we withdraw the claim that it surpasses the teacher. At fixed memory, C3 recovers most of the teacher’s retrieval gain on NIAH multikey. At 32K, accuracy is .948 for C3, .985 for the full-history teacher, and .062 for the live window alone.
W2 / Q1. Direct comparisons with CCM and Still. We tested CCM-style averaging within C3 and found that retaining whole rows outperformed averaging the old and new states. The published CCM setting writes 2–8 compression tokens per step, while matching our state directly would require about 2,048. Still compacts an individual context and keeps each compacted chunk in its multi-cycle version, making it not bounded state sized for the continual version of this method.
W3 / Q4. Scale and an in-budget retention baseline. We evaluated continual GD performance at 1.7B on 8K–16K streams. At 8B we measured cost only. Knowledge Seeding takes 1,014 seconds to consolidate one 32K stream on an H100, against 4.4 seconds for C3 (Fig. 8). At 1.7B, the oracle row-state experiment separates acquisition from retention. GD answers .998 of the newest block’s questions correctly just after fitting them with gold supervision. One write later, GD accuracy on those questions is at most .02, compared with .435 for C3 (Fig. 2).
Knowledge Seeding without replay carries its updated adapter from block to block and fits the deployment budget. C3 leads the in-budget variant on all three tasks at 8K, including .766 versus .393 on TriviaQA (Fig. 13, App. E). We also tested a version that retains all prior quiz pairs and excluded that buffer from its budget, giving GD extra storage, which only improved performance slightly.
Q2. Intervals and training seeds. The ablations use a single training seed. Figures 4 and 12 report two trained state sizes and 14 tasks, most at eight context lengths. The advantage of C3 over full-history ICL on TREC did not carry over to the revised merge rule, we update the statement.
Q3. Exposure to evaluation tasks. The revised C3 trains on questions generated about general long documents. The revised banking77 and NIAH results come from a model trained without either evaluation task. The submitted model did receive task data in its decay stage, including banking77 demonstrations, but never the held out test questions.
Q4. YaRN at 131K. The submitted EventQA comparison used a factor of 2 for C3 and 4 for the baselines. We now use 4 for all methods. Matching the factors changes C3’s accuracy by .014. The number of retained rows is fixed, but the rows keep their original rotary positions, so their distances to a query on the live window grow with the stream.
Overall, thank you for your feedback and interest in our paper! We hope to have addressed all of you questions, and would appreciate your reconsideration of the score. Please let us know if you have further comments or questions.