- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether off-the-shelf LLMs can use information encoded into their weights through test-time updates as context, without additional meta-training. It proposes Addressable Context Encoding (ACE), which associates each context chunk with a unique identifier to facilitate recall and introduces a prompting framework to guide the use of the recalled information as context. Experiments on controlled tasks and long-context benchmarks show improvements over basic autoregressive context encoding.
Critical strengths
- The paper identifies a clear and relevant distinction between memorizing information in model weights and using it as context, and studies it with off-the-shelf LLMs to avoid the confounding effect of meta-training.
- ACE requires no architectural changes or additional training beyond test-time updates, yet generally improves over basic autoregressive encoding across multiple models and tasks. The controlled sandbox tasks are designed to evaluate distinct aspects of context use, including recalling encoded information, using it in subsequent requests, and selecting between encoded information and prior knowledge.
Critical weaknesses
W1: Limited baselines. (1) The only encoding baseline is basic autoregressive encoding. Within the paper's setting of no task-specific meta-training, a stronger encoding objective such as Deep Context Distillation [1] without its task-specific Knowledge Extractor would help assess whether ACE's benefits extend beyond the chosen baseline. Although Sections 2.3 and 5 briefly discuss practical limitations and computational costs of context distillation, no empirical comparison is provided to support these claims. (2) The paper does not report a no-context baseline, i.e., the original model answering without the context or any encoding. This is important on LongBench, where questions come from real documents and may be partly answerable from prior knowledge. In Table 3, both TCE and ACE score far below full-context inference, and ACE improves over TCE by only 0.2–1.9 F1 on the Qwen3.5 models. Without such a baseline, it is unclear how much of their performance comes from the information encoded into the weights rather than from the model's prior knowledge.
W2: Source selection remains unreliable. ACE performs well when explicitly asked to answer from the encoded context, but two-sided Counterfactual accuracy is only 20–36.5% (Table 2), and recovery of prior knowledge after encoding is only 25.5–39% (Table 19). These results indicate that the model cannot reliably switch between encoded information and prior knowledge as requested. While the paper acknowledges lower two-sided accuracy, the claim in the abstract and introduction that ACE makes encoded information distinguishable from pre-existing knowledge appears overstated.
[1] Caccia, Lucas, et al. "Training plug-n-play knowledge modules with deep context distillation." arXiv preprint arXiv:2503.08727 (2025).
Questions for authors
Q1: The paper argues that meta-training confounds the gains of prior TCE methods. Is there empirical evidence for this, for example by evaluating a meta-trained method on tasks outside its meta-training distribution?
Q2: Tables 5 and 21–22 leave the respective contributions of UID conditioning, encoding format, and explicit invocation unclear. Could the authors clarify which results specifically establish the contribution of UID-based addressability? For example, a baseline encoding the context with the same conversational framing but without identifiers would help isolate this contribution. In addition, since the sandbox tasks use a single chunk, could the authors evaluate selective access when multiple independent contexts are encoded into the same model state and only one is invoked by its UID?
Q3: On LongBench v2, even full-context inference obtains only 2.5–4.0 out of 100 (Table 3) under the judge-based evaluation. Were the answer options provided to the evaluated model, or only to the judge? Could the authors explain this low performance and clarify the evaluation protocol? This affects whether the comparison between TCE and ACE on LongBench v2 is informative.
Q4: In the main experiments, the contexts fit within the models' context windows, and full-context inference is generally more accurate. Appendix F also shows substantial encoding overhead. Under what practical conditions would ACE offer a favorable accuracy–cost trade-off? Could the authors evaluate ACE on contexts exceeding the evaluated models' native context windows, where full-context inference is infeasible, and compare it with retrieval-, compression-, or truncation-based alternatives? The longest-context evaluation in Appendix E.4 uses 64K contexts, which remain within the native context window (Table 31), and covers only NIAH-1. Please also report the number of evaluated instances.
Q5: Is ACE complementary to meta-training? For example, does applying ACE's encoding objectives and invocation prompt to a meta-trained model further improve its performance?
10 comments
Response to Reviewer 6Z3e (2/4)
Q3 (W2). Source selection remains unreliable. Two-sided Counterfactual accuracy is only 20–36.5% (Table 2), and recovery of prior knowledge after encoding is only 25.5–39% (Table 19). The claim that ACE makes encoded information distinguishable from pre-existing knowledge appears overstated.
A3. We would like to clarify that our claim is made relative to TCE, and our results provide convincing support for it. Compared with TCE, ACE makes the encoded information much easier to tell apart from pre-existing knowledge, and raises Counterfactual accuracy from 3.5–14.0% to over 92%:
| LLM | Counterfactual (TCE) | Counterfactual (ACE) | Two-sided (TCE) | Two-sided (ACE) |
|---|---|---|---|---|
| Qwen3.5-4B | 3.5 | 92.5 | 0.5 | 27.5 |
| Qwen3.5-9B | 14.0 | 94.5 | 6.0 | 30.5 |
| Qwen3.5-27B | 11.0 | 93.0 | 8.0 | 29.0 |
Two-sided source selection is a much stricter test. For the same encoded fact, the model has to answer from the encoded context for one query, and ignore it for another. Even here, where TCE almost completely fails (0.5–8.0%), ACE improves accuracy by more than 3× on every model. Recovering prior knowledge after encoding (Table 19) is similarly strict.
Note that directly comparing accuracy across subtasks is not fair, since it ignores the difficulty of each task itself. The lower absolute accuracy on these stricter tests reflects their difficulty, rather than unreliable source selection. We therefore respectfully disagree that the claim is overstated, and we will further clarify it in our revision.
Q4 (Q1). The paper argues that meta-training confounds the gains of prior TCE methods. Is there empirical evidence for this, e.g., by evaluating a meta-trained method on tasks outside its meta-training distribution?
A4. Directly verifying this is beyond our scope and resources. Meta-trained methods such as Titans (Behrouz et al., 2025b) require large-scale training, which is mostly feasible only for industrial labs.
Nevertheless, the problem can already be seen from their experimental settings, where meta-training and evaluation data come from the same task distribution. Take RECKONING (Chen et al., 2023) and PERK (Chen et al., 2026), the works most closely related to ours. RECKONING is meta-trained on the training splits of ProofWriter and CLUTRR-SG and evaluated on their test splits. PERK is meta-trained and evaluated on the same synthetic Student Records task, where only the entities and attribute combinations differ.
In such settings, the model can overfit to task-specific patterns during meta-training, such as question formats and answer types, so the reported gains cannot be credited to context encoding alone. To our knowledge, none of these works reports an ablation that separates the two. This is also beyond our scope, so we take it as an important direction for future work.
Response to Reviewer 6Z3e (3/4)
Q5 (Q2). Tables 5 and 21–22 leave the contributions of UID conditioning, encoding format, and explicit invocation unclear. Which results establish the contribution of UID-based addressability? A baseline with the same conversational framing but without identifiers would help. Could the authors also evaluate selective access when multiple independent contexts are encoded into the same model state and only one is invoked by its UID?
A5. We have examined a closely related configuration in Appendix D.5, where the conversational framing is kept but every UID is replaced by the shared string "memory", so that the chunks no longer have distinct identifiers. Unique UIDs outperform the shared one on all models (Table 21):
| Recognition | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B |
|---|---|---|---|
| ACE with shared UID | 66.0 | 91.9 | 47.7 |
| ACE | 81.7 | 95.4 | 58.4 |
Moreover, with the encoded model fixed, replacing the correct UID with a decoy of the same length makes NIAH-1 accuracy drop sharply, which shows that access depends on the specific UID (Table 30):
| NIAH-1 | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B |
|---|---|---|---|
| Decoy UID | 1.0 | 8.0 | 4.0 |
| Correct UID | 87.0 | 84.0 | 100.0 |
We have also examined the encoding format and the contributions of encoding and invocation (Table 22 and Tables 4–5), which show that the gains of ACE do not depend on a specific wording and that both components contribute.
Regarding selective access, our long-context experiments already encode multiple chunks with distinct UIDs into the same model (Table 3). In addition, following your suggestion, we further jointly encode = 2, 3, or 4 independent records from our Recall set into the same weights, and then query one record by its UID. An answer is counted as correct only if it contains the requested value and no value from the other records. The accuracy (%) of TCE / ACE is reported below:
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | |
|---|---|---|---|
| 2 | 0.5 / 53.1 | 1.0 / 38.3 | 2.0 / 48.5 |
| 3 | 2.6 / 50.8 | 0.5 / 36.9 | 2.1 / 41.0 |
| 4 | 2.1 / 49.5 | 0.5 / 39.6 | 0.5 / 38.0 |
As we can see, ACE remains effective when multiple independent records are encoded into the same model, while TCE stays near zero. Moreover, ACE rarely confuses records: answers containing another record's value account for only 3–13% of all queries. Thank you for this suggestion, and we will add this experiment in the revision.
Q6 (Q3). On LongBench v2, even full-context inference obtains only 2.5–4.0 out of 100 under the judge-based evaluation. Were the answer options provided to the evaluated model, or only to the judge? Please explain this low performance and clarify the evaluation protocol.
A6. The answer options were given to both the evaluated model and the judge. The model received the question with the four options written as prose, without option letters or an explicit instruction to choose one, and generated a free-form answer. The judge then scored each answer as 1 (correct), 0.5 (partially correct), or 0 (otherwise, including answers that never commit to an option), without seeing the method name or the source document.
The low scores come from two factors. First, under this free-form protocol, many responses never commit to an option and get 0 points. Second, LongBench v2 is hard by design, since a question was dropped if all three reference LLMs could answer it (Bai et al., 2025). The same protocol is applied to all methods, including full-context inference, so the comparison remains fair. We will clarify this protocol in the revision.
Response to Reviewer 6Z3e (1/4)
We sincerely thank the reviewer for the thoughtful feedback. We understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.
A1. We would first note that TCE is a fairly new direction, and in our setting only autoregressive TCE is directly comparable. Related directions such as key-value association and context distillation are fundamentally different from ours. Key-value association can be viewed as a form of linear attention (Liu et al., 2026), and context distillation trains the model to generalize from representative context–query pairs (Caccia et al., 2025). In contrast, our goal is to encode the context itself into the model weights, so that relevant information can be flexibly recalled for any later request.
The most related works, RECKONING (Chen et al., 2023) and PERK (Chen et al., 2026), use the same autoregressive encoding as our TCE baseline, but additionally require meta-training. Since ACE is built directly on top of autoregressive TCE, TCE is the most direct and fair baseline for us to compare with. We admit that the current version does not make the novelty of our problem setting clear enough, and we will clarify it in the revision.
We agree that the comparison you suggest is fair and interesting. Following your suggestion, we further compare ACE with Deep Context Distillation (DCD) without its task-specific Knowledge Extractor, including both of its variants: DDCD, which distills on the document itself, and SDCD, which distills on model-generated summaries. The accuracy (%) on Recall / Recognition is reported below (ACE with ci_v1):
As we can see, both DCD variants stay at the level of TCE and far below ACE. So the benefits of ACE extend beyond the autoregressive baseline.
A2. For the sandbox tasks, the no-context baseline is controlled by construction. Before encoding, we tested every candidate on the unmodified models, and kept it only if the model failed without the context but succeeded with the full context. So the no-context accuracy is 0% on all kept instances. The only exception is Outdated Information, which is built per model from each model's own outdated beliefs. We have mentioned this screening in Section 4.1, and will further clarify it in the revision.
For LongBench and LongBench v2, following your suggestion, we report the original models answering without the context or any encoding. LongBench (F1):
LongBench v2:
As we can see, on LongBench TCE stays at the no-context level, which means it makes little use of the encoded information. In contrast, ACE improves over the no-context baseline on all four models and on both benchmarks.