acceptodds
Under review as a conference paper at ICLR 2027

Encoding Context into LLM Weights: What Makes It Context?

Abstract

Longer contexts allow large language models (LLMs) to condition their predictions on more information, but inference costs grow prohibitively with context length. An emerging alternative uses test-time updates to encode context as temporary changes to model weights. However, encoding this information into weights that already represent general knowledge obscures its role as context, so the LLM may fail to use it properly to condition subsequent inference. To restore this contextual role, we first associate each context with an identifier during encoding, thereby distinguishing the newly encoded information from pre-existing knowledge. Invoking the identifier at inference then enables the LLM to recite the encoded context. Nevertheless, successful recitation does not ensure that the LLM will use the recalled information as context. We therefore further introduce a prompting framework that explicitly guides the LLM to use this information as context. Across diverse controlled tasks and practical benchmarks, our method outperformed the baseline using only off-the-shelf LLMs, showing its advances in context encoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM memory, context utilization, large language models, test-time training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6Z3eOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether off-the-shelf LLMs can use information encoded into their weights through test-time updates as context, without additional meta-training. It proposes Addressable Context Encoding (ACE), which associates each context chunk with a unique identifier to facilitate recall and introduces a prompting framework to guide the use of the recalled information as context. Experiments on controlled tasks and long-context benchmarks show improvements over basic autoregressive context encoding.

Critical strengths
  • The paper identifies a clear and relevant distinction between memorizing information in model weights and using it as context, and studies it with off-the-shelf LLMs to avoid the confounding effect of meta-training.
  • ACE requires no architectural changes or additional training beyond test-time updates, yet generally improves over basic autoregressive encoding across multiple models and tasks. The controlled sandbox tasks are designed to evaluate distinct aspects of context use, including recalling encoded information, using it in subsequent requests, and selecting between encoded information and prior knowledge.
Critical weaknesses

W1: Limited baselines. (1) The only encoding baseline is basic autoregressive encoding. Within the paper's setting of no task-specific meta-training, a stronger encoding objective such as Deep Context Distillation [1] without its task-specific Knowledge Extractor would help assess whether ACE's benefits extend beyond the chosen baseline. Although Sections 2.3 and 5 briefly discuss practical limitations and computational costs of context distillation, no empirical comparison is provided to support these claims. (2) The paper does not report a no-context baseline, i.e., the original model answering without the context or any encoding. This is important on LongBench, where questions come from real documents and may be partly answerable from prior knowledge. In Table 3, both TCE and ACE score far below full-context inference, and ACE improves over TCE by only 0.2–1.9 F1 on the Qwen3.5 models. Without such a baseline, it is unclear how much of their performance comes from the information encoded into the weights rather than from the model's prior knowledge.

W2: Source selection remains unreliable. ACE performs well when explicitly asked to answer from the encoded context, but two-sided Counterfactual accuracy is only 20–36.5% (Table 2), and recovery of prior knowledge after encoding is only 25.5–39% (Table 19). These results indicate that the model cannot reliably switch between encoded information and prior knowledge as requested. While the paper acknowledges lower two-sided accuracy, the claim in the abstract and introduction that ACE makes encoded information distinguishable from pre-existing knowledge appears overstated.

[1] Caccia, Lucas, et al. "Training plug-n-play knowledge modules with deep context distillation." arXiv preprint arXiv:2503.08727 (2025).

Questions for authors

Q1: The paper argues that meta-training confounds the gains of prior TCE methods. Is there empirical evidence for this, for example by evaluating a meta-trained method on tasks outside its meta-training distribution?

Q2: Tables 5 and 21–22 leave the respective contributions of UID conditioning, encoding format, and explicit invocation unclear. Could the authors clarify which results specifically establish the contribution of UID-based addressability? For example, a baseline encoding the context with the same conversational framing but without identifiers would help isolate this contribution. In addition, since the sandbox tasks use a single chunk, could the authors evaluate selective access when multiple independent contexts are encoded into the same model state and only one is invoked by its UID?

Q3: On LongBench v2, even full-context inference obtains only 2.5–4.0 out of 100 (Table 3) under the judge-based evaluation. Were the answer options provided to the evaluated model, or only to the judge? Could the authors explain this low performance and clarify the evaluation protocol? This affects whether the comparison between TCE and ACE on LongBench v2 is informative.

Q4: In the main experiments, the contexts fit within the models' context windows, and full-context inference is generally more accurate. Appendix F also shows substantial encoding overhead. Under what practical conditions would ACE offer a favorable accuracy–cost trade-off? Could the authors evaluate ACE on contexts exceeding the evaluated models' native context windows, where full-context inference is infeasible, and compare it with retrieval-, compression-, or truncation-based alternatives? The longest-context evaluation in Appendix E.4 uses 64K contexts, which remain within the native context window (Table 31), and covers only NIAH-1. Please also report the number of evaluated instances.

Q5: Is ACE complementary to meta-training? For example, does applying ACE's encoding objectives and invocation prompt to a meta-trained model further improve its performance?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 6Z3e (1/4)

We sincerely thank the reviewer for the thoughtful feedback. We understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.

Q1 (W1, part 1). The only encoding baseline is basic autoregressive encoding. A stronger objective such as Deep Context Distillation (Caccia et al., 2025) without its task-specific Knowledge Extractor would help assess whether ACE's benefits extend beyond the chosen baseline. Sections 2.3 and 5 discuss practical limitations and costs of context distillation, but no empirical comparison is provided.

A1. We would first note that TCE is a fairly new direction, and in our setting only autoregressive TCE is directly comparable. Related directions such as key-value association and context distillation are fundamentally different from ours. Key-value association can be viewed as a form of linear attention (Liu et al., 2026), and context distillation trains the model to generalize from representative context–query pairs (Caccia et al., 2025). In contrast, our goal is to encode the context itself into the model weights, so that relevant information can be flexibly recalled for any later request.

The most related works, RECKONING (Chen et al., 2023) and PERK (Chen et al., 2026), use the same autoregressive encoding as our TCE baseline, but additionally require meta-training. Since ACE is built directly on top of autoregressive TCE, TCE is the most direct and fair baseline for us to compare with. We admit that the current version does not make the novelty of our problem setting clear enough, and we will clarify it in the revision.

We agree that the comparison you suggest is fair and interesting. Following your suggestion, we further compare ACE with Deep Context Distillation (DCD) without its task-specific Knowledge Extractor, including both of its variants: DDCD, which distills on the document itself, and SDCD, which distills on model-generated summaries. The accuracy (%) on Recall / Recognition is reported below (ACE with ci_v1):

MethodQwen3.5-4BQwen3.5-9B
TCE6.1 / 0.54.6 / 2.0
DDCD5.1 / 0.07.1 / 0.0
SDCD8.1 / 0.012.2 / 0.0
ACE55.3 / 81.735.5 / 95.4

As we can see, both DCD variants stay at the level of TCE and far below ACE. So the benefits of ACE extend beyond the autoregressive baseline.

Q2 (W1, part 2). No no-context baseline is reported, i.e., the original model answering without the context or any encoding. This matters on LongBench, where questions may be partly answerable from prior knowledge. ACE improves over TCE by only 0.2–1.9 F1 on the Qwen3.5 models, and it is unclear how much performance comes from the encoded information rather than prior knowledge.

A2. For the sandbox tasks, the no-context baseline is controlled by construction. Before encoding, we tested every candidate on the unmodified models, and kept it only if the model failed without the context but succeeded with the full context. So the no-context accuracy is 0% on all kept instances. The only exception is Outdated Information, which is built per model from each model's own outdated beliefs. We have mentioned this screening in Section 4.1, and will further clarify it in the revision.

For LongBench and LongBench v2, following your suggestion, we report the original models answering without the context or any encoding. LongBench (F1):

LLMNo contextTCEACEFull context
Qwen3.5-4B8.48.89.020.8
Qwen3.5-9B8.98.29.620.3
Qwen3.5-27B7.97.99.818.9
Muse-Glimmer-30B15.214.019.221.2

LongBench v2:

LLMNo contextTCEACEFull context
Qwen3.5-4B0.00.02.02.5
Qwen3.5-9B1.52.53.53.0
Qwen3.5-27B1.02.52.54.0
Muse-Glimmer-30B1.02.03.03.0

As we can see, on LongBench TCE stays at the no-context level, which means it makes little use of the encoded information. In contrast, ACE improves over the no-context baseline on all four models and on both benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 6Z3e (2/4)

Q3 (W2). Source selection remains unreliable. Two-sided Counterfactual accuracy is only 20–36.5% (Table 2), and recovery of prior knowledge after encoding is only 25.5–39% (Table 19). The claim that ACE makes encoded information distinguishable from pre-existing knowledge appears overstated.

A3. We would like to clarify that our claim is made relative to TCE, and our results provide convincing support for it. Compared with TCE, ACE makes the encoded information much easier to tell apart from pre-existing knowledge, and raises Counterfactual accuracy from 3.5–14.0% to over 92%:

LLMCounterfactual (TCE)Counterfactual (ACE)Two-sided (TCE)Two-sided (ACE)
Qwen3.5-4B3.592.50.527.5
Qwen3.5-9B14.094.56.030.5
Qwen3.5-27B11.093.08.029.0

Two-sided source selection is a much stricter test. For the same encoded fact, the model has to answer from the encoded context for one query, and ignore it for another. Even here, where TCE almost completely fails (0.5–8.0%), ACE improves accuracy by more than 3× on every model. Recovering prior knowledge after encoding (Table 19) is similarly strict.

Note that directly comparing accuracy across subtasks is not fair, since it ignores the difficulty of each task itself. The lower absolute accuracy on these stricter tests reflects their difficulty, rather than unreliable source selection. We therefore respectfully disagree that the claim is overstated, and we will further clarify it in our revision.

Q4 (Q1). The paper argues that meta-training confounds the gains of prior TCE methods. Is there empirical evidence for this, e.g., by evaluating a meta-trained method on tasks outside its meta-training distribution?

A4. Directly verifying this is beyond our scope and resources. Meta-trained methods such as Titans (Behrouz et al., 2025b) require large-scale training, which is mostly feasible only for industrial labs.

Nevertheless, the problem can already be seen from their experimental settings, where meta-training and evaluation data come from the same task distribution. Take RECKONING (Chen et al., 2023) and PERK (Chen et al., 2026), the works most closely related to ours. RECKONING is meta-trained on the training splits of ProofWriter and CLUTRR-SG and evaluated on their test splits. PERK is meta-trained and evaluated on the same synthetic Student Records task, where only the entities and attribute combinations differ.

In such settings, the model can overfit to task-specific patterns during meta-training, such as question formats and answer types, so the reported gains cannot be credited to context encoding alone. To our knowledge, none of these works reports an ablation that separates the two. This is also beyond our scope, so we take it as an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 6Z3e (3/4)

Q5 (Q2). Tables 5 and 21–22 leave the contributions of UID conditioning, encoding format, and explicit invocation unclear. Which results establish the contribution of UID-based addressability? A baseline with the same conversational framing but without identifiers would help. Could the authors also evaluate selective access when multiple independent contexts are encoded into the same model state and only one is invoked by its UID?

A5. We have examined a closely related configuration in Appendix D.5, where the conversational framing is kept but every UID is replaced by the shared string "memory", so that the chunks no longer have distinct identifiers. Unique UIDs outperform the shared one on all models (Table 21):

RecognitionQwen3.5-4BQwen3.5-9BQwen3.5-27B
ACE with shared UID66.091.947.7
ACE81.795.458.4

Moreover, with the encoded model fixed, replacing the correct UID with a decoy of the same length makes NIAH-1 accuracy drop sharply, which shows that access depends on the specific UID (Table 30):

NIAH-1Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Decoy UID1.08.04.0
Correct UID87.084.0100.0

We have also examined the encoding format and the contributions of encoding and invocation (Table 22 and Tables 4–5), which show that the gains of ACE do not depend on a specific wording and that both components contribute.

Regarding selective access, our long-context experiments already encode multiple chunks with distinct UIDs into the same model (Table 3). In addition, following your suggestion, we further jointly encode KK = 2, 3, or 4 independent records from our Recall set into the same weights, and then query one record by its UID. An answer is counted as correct only if it contains the requested value and no value from the other records. The accuracy (%) of TCE / ACE is reported below:

KKQwen3.5-4BQwen3.5-9BQwen3.5-27B
20.5 / 53.11.0 / 38.32.0 / 48.5
32.6 / 50.80.5 / 36.92.1 / 41.0
42.1 / 49.50.5 / 39.60.5 / 38.0

As we can see, ACE remains effective when multiple independent records are encoded into the same model, while TCE stays near zero. Moreover, ACE rarely confuses records: answers containing another record's value account for only 3–13% of all queries. Thank you for this suggestion, and we will add this experiment in the revision.

Q6 (Q3). On LongBench v2, even full-context inference obtains only 2.5–4.0 out of 100 under the judge-based evaluation. Were the answer options provided to the evaluated model, or only to the judge? Please explain this low performance and clarify the evaluation protocol.

A6. The answer options were given to both the evaluated model and the judge. The model received the question with the four options written as prose, without option letters or an explicit instruction to choose one, and generated a free-form answer. The judge then scored each answer as 1 (correct), 0.5 (partially correct), or 0 (otherwise, including answers that never commit to an option), without seeing the method name or the source document.

The low scores come from two factors. First, under this free-form protocol, many responses never commit to an option and get 0 points. Second, LongBench v2 is hard by design, since a question was dropped if all three reference LLMs could answer it (Bai et al., 2025). The same protocol is applied to all methods, including full-context inference, so the comparison remains fair. We will clarify this protocol in the revision.

Reviewer_hVq5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Addressable Context Encoding (ACE), which associates each context chunk with a unique identifier through temporary weight updates. A prompting framework then guides the model to access and use the encoded information during inference. The authors evaluate ACE on controlled tasks covering different forms of context use. The results show improvements over standard TCE on several tasks. The paper also provides mathematical analysis to explain potential benefits of UID conditioning.

Critical strengths

ACE offers a simple way to improve context use in off-the-shelf LLMs without additional meta-training. The authors test ACE on several context-related tasks. Ablation studies help explain the encoding and prompting. These experiments support the approach. The mathematical analysis also provides additional insight.

Critical weaknesses

My main concern is that ACE's context capacity remains unclear. The paper does not establish how much context a model can reliably encode. The baseline comparison is also limited, with TCE as the main encoding baseline. This makes it difficult to assess ACE's benefits over other relevant methods. Finally, ACE requires a new round of weight updates for each independent context. The paper reports computational costs, but its practical efficiency benefits remain unclear.

Questions for authors
  1. ACE updates model parameters to encode context. Could the authors analyze how much context a model can reliably encode with ACE?
  2. The comparison mainly focuses on TCE. Could the authors include comparisons with other relevant context-encoding methods?
  3. The MMLU-200 evaluation does not include the unmodified model as a baseline. Could the authors add this comparison to assess changes in the model's original capabilities?
  4. The cost measurements use one fixed example per benchmark family. This provides limited empirical evidence on how ACE's costs scale with context length. Could the authors measure how its encoding and inference costs change as context length increases?
11 comments
AuthorsOpenReview ↗

Response to Reviewer hVq5 (1/2)

We sincerely thank the reviewer for the thoughtful feedback. We understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.

Q1 (W & Q1). ACE's context capacity remains unclear. Could the authors analyze how much context a model can reliably encode with ACE?

A1. We would like to clarify that how much information can be stored and whether it can be reliably used as context are two quite different questions, and our focus is the latter. If the model cannot use the encoded context, encoding more of it brings no benefit. TCE is a good example: it does encode the context into its weights, but largely fails to use it (Tables 1–2). With this focus, our experiments already cover context lengths from 1K to 64K tokens. For example, on Qwen3.5-9B, ACE outperforms TCE at every length:

BenchmarkContext lengthTCEACE
RULER NIAH-21K7.017.0
RULER NIAH-24K6.716.7
LongBench2–7K8.29.6
LongBench v210–15K2.53.5
RULER NIAH-164K30.080.0

Moreover, the capacity of context encoding is not specific to ACE, but mainly depends on the fast weights that store the context. Intuitively, using more parameters as fast weights means a larger capacity. By default we update only the final FFN layer, and updating more layers can further improve performance. For example, ATTN-3 increases Continuation accuracy from 48.8% to 72.0% on Qwen3.5-4B (Appendix C.3). The specific design of fast weights is beyond the scope of this paper, and we leave it to future work.

Q2 (W & Q2). The baseline comparison is limited, with TCE as the main encoding baseline. Could the authors include comparisons with other relevant context-encoding methods?

A2. We would first note that TCE is a fairly new direction, and in our setting only autoregressive TCE is directly comparable. Although we discuss key-value association (e.g., Titans; Behrouz et al., 2025b) and context distillation as related directions, they are fundamentally different from ours. Key-value association can be viewed as a form of linear attention (Liu et al., 2026), and context distillation trains the model to generalize from context–query pairs (Caccia et al., 2025). In contrast, our goal is to encode the context itself into the model weights, so that relevant information can be flexibly recalled for any later request.

The most related works are RECKONING (Chen et al., 2023) and PERK (Chen et al., 2026). They use the same autoregressive encoding as ours, i.e., our TCE baseline, but additionally require meta-training, while ACE requires none. As discussed in Section 2.3, meta-training may introduce reliance on task-specific correlations, which makes it hard to separate its contribution from that of TCE itself. Since ACE is built directly on top of TCE, TCE is the most direct and fair baseline for us to compare with.

In fact, very few works have examined the limitations of TCE itself, and there is not even a consensus on the name TCE. We admit that the current version does not sufficiently emphasize the novelty of our setting, and discusses TCE together with other directions. We will clarify this in the revision.

Moreover, following your suggestion, we further compare ACE with Deep Context Distillation (DCD; Caccia et al., 2025) without its task-specific Knowledge Extractor, including both of its variants: DDCD, which distills on the document itself, and SDCD, which distills on model-generated summaries. The accuracy (%) on Recall / Recognition is reported below (ACE with ci_v1):

MethodQwen3.5-4BQwen3.5-9B
TCE6.1 / 0.54.6 / 2.0
DDCD5.1 / 0.07.1 / 0.0
SDCD8.1 / 0.012.2 / 0.0
ACE55.3 / 81.735.5 / 95.4

As we can see, both DCD variants stay at the level of TCE and far below ACE.

AuthorsOpenReview ↗

Response to Reviewer hVq5 (2/2)

Q3 (Q3). The MMLU-200 evaluation does not include the unmodified model as a baseline. Could the authors add this comparison to assess changes in the model's original capabilities?

A3. Following your suggestion, we evaluate the original models on the same MMLU-200 questions, and will add them in our revision. The table below reports MMLU-200 accuracy (%) at the default threshold τ=0.1\tau = 0.1, averaged over the three RULER tasks used for encoding:

LLMOriginalTCEACE (ci_short)ACE (ci_v1)ACE (ci_v2)
Qwen3.5-4B76.074.174.573.072.1
Qwen3.5-9B77.576.371.768.365.8
Qwen3.5-27B81.579.981.880.880.3
Muse-Glimmer-30B74.069.576.275.376.7

As we can see, in most cases encoding changes MMLU-200 accuracy only moderately. Also note that encoding only updates the fast weights temporarily, and the original weights are fully restored once the context is no longer needed (Section 3.3).

Q4 (W & Q4). ACE requires a new round of weight updates for each independent context, and its practical efficiency benefits remain unclear. The cost measurements use one fixed example per benchmark family. Could the authors measure how encoding and inference costs change as context length increases?

A4. First, the encoding cost of ACE grows linearly with context length, since the chunk size is fixed and the number of updates per chunk is bounded, whereas the attention cost of full-context inference grows quadratically (Appendix F). Moreover, encoding is done only once per context and is then reused by all later queries. So by its mechanism, ACE scales better to long contexts, which is exactly why we study this direction.

Our measurements agree with this. From the LongBench example to the longer LongBench v2 example, the encoding throughput of ACE stays nearly constant on all four models, so the encoding time grows in proportion to the context length (Tables 33–36):

Encoding throughput of ACE (tokens/s)Qwen3.5-4BQwen3.5-9BQwen3.5-27BMuse-Glimmer-30B
LongBench362365145503
LongBench v2 (longer)362367158521

Second, at inference, ACE no longer needs to process the original context, since it is removed after encoding, whereas full-context inference has to process the entire context for every query. On the LongBench v2 example, ACE has lower response latency than full-context inference on all four models (Tables 33–36), and uses about 20–24× fewer FLOPs per query and 22–27× less KV-cache storage (Appendix F):

Response latency on LongBench v2 (s)Qwen3.5-4BQwen3.5-9BQwen3.5-27BMuse-Glimmer-30B
Full context7.377.7817.2013.47
ACE4.925.1510.175.44

On the three Qwen3.5 models, both methods generate the same number of output tokens (128). On Muse-Glimmer-30B, ACE generates a shorter response (95 tokens), which also contributes to its lower latency.

We have tried our best to address your concerns and would like to know whether our responses have addressed them. If anything remains unclear, we are more than happy to discuss it further.

Reviewer_2qweOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to encode contextual information into LLM weights so that models can answer questions after the original context is removed. The authors distinguish memorizing information from correctly using it as context. They propose Addressable Context Encoding (ACE), which combines UID-conditioned encoding to establish access cues with role-binding text that specifies how the encoded information should function as context. At inference time, prompts containing the UIDs guide the model to use this information. ACE operates on off-the-shelf models through test-time updates, without additional task-specific meta-training. Experiments cover factual recall, complete-statement reconstruction, conflicts between context and prior knowledge, NIAH retrieval, and LongBench document question answering. The results show improvements over standard context encoding across multiple tasks, while substantial limitations remain in information-source selection and complex long-document question answering.

Critical strengths

Evaluation extends to more practical retrieval and document QA tasks. The paper extends its evaluation beyond isolated facts to retrieval amid distracting text in NIAH and long-document question answering in LongBench and LongBench v2. These settings more closely reflect practical needs to locate and use contextual information. Comparisons with full-context inference also help assess how much useful information remains accessible after encoding.

Critical weaknesses
  1. The gains without UID invocation remain insufficiently explained. Tables 4 and 5 report Qwen3.5-9B Recall accuracies of 4.6% for standard TCE (Table 5, line 5), 1.5% for UID-conditioned encoding alone (Table 4, line 6), and 19.8% for full ACE encoding (Table 5, line 8), all without UIDs or invocation prompts at inference. Full ACE improves over TCE by 15.2 percentage points, while UID-conditioned encoding alone provides no corresponding benefit. The proposed memorization benefit of UID conditioning therefore does not adequately explain this result. Although the authors acknowledge the phenomenon, they do not sufficiently disentangle its causes. The theoretical analysis primarily concerns loss reduction during UID-conditioned training and does not explain improved question answering after removing the UID. This substantial gap suggests that some gains may arise from training effects beyond addressability, such as repeated exposure to the same text under two different prefixes. The authors should isolate each encoding component’s contribution under UID-free inference, with matched text exposure and optimization budgets, to clarify the mechanisms underlying the gains and their relationship to the claimed contribution.
  2. The effect of explicit context-use instructions remains unclear. Qwen3.5-9B achieves 94.5% accuracy on Counterfactual (Table 2, row 7), where queries explicitly instruct the model to use only the context, even when it contradicts prior knowledge. How much does this high accuracy depend on that instruction? Would adding the same instruction to Recall substantially improve its 35.5% accuracy (Table 1, row 7)? The authors should compare prompts with and without this instruction while holding the encoded weights and evaluation samples fixed, to clarify whether more explicit prompting improves access to the encoded information.
  3. Some long-context results require further analysis, and their statistical reliability is unclear. In Table 3, Qwen3.5-9B achieves 3.5 points with ACE on LongBench v2, exceeding the 3.0 points obtained with full-context inference (rows 6 and 4). This is surprising and warrants analysis, particularly given the very low scores out of 100 and the small difference of 0.5 points. The authors should examine individual examples to explain why ACE performs better in these cases. Similarly, the LongBench improvement from 8.2 to 9.6 F1 (rows 5 and 6) is reported without variability estimates, confidence intervals, or significance tests. Paired sample-level analysis and uncertainty estimates would help assess whether these gains are reliable.
Questions for authors

No further questions. See Weaknesses.

12 comments
AuthorsOpenReview ↗

Response to Reviewer 2qwe (1/3)

We sincerely thank the reviewer for the thoughtful feedback and the detailed suggestions. We understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.

Q1 (W1). The gains without UID invocation remain insufficiently explained. Without UIDs at inference, Qwen3.5-9B Recall accuracy is 4.6% for TCE, 1.5% for UID-conditioned encoding alone (Table 4), and 19.8% for full ACE encoding (Table 5). UID conditioning alone provides no benefit, and the theoretical analysis concerns loss reduction during UID-conditioned training rather than question answering after removing the UID. Some gains may arise from training effects beyond addressability, such as repeated exposure to the same text under two different prefixes. Please isolate each encoding component's contribution under UID-free inference, with matched text exposure and optimization budgets.

A1. We would first note that the UID-free gain is expected under our design. ACE jointly optimizes LA\mathcal{L}_A and LB\mathcal{L}_B during encoding (Eq. 10), and the two are designed to work together rather than separately. In fact, our ablations on Qwen3.5-9B Recall (Tables 4–5) already separate their contributions:

EncodingInference cueRecall
TCE (L\mathcal{L})–4.6
LA\mathcal{L}_A only–1.5
LA\mathcal{L}_A onlyUID0.5
LA\mathcal{L}_A onlyACE inference22.8
LB\mathcal{L}_B onlyACE inference7.6
LA+LB\mathcal{L}_A + \mathcal{L}_B–19.8
LA+LB\mathcal{L}_A + \mathcal{L}_BACE inference35.5

This matches our design rationale: off-the-shelf LLMs have never been trained for test-time encoding or seen our UIDs, so LA\mathcal{L}_A alone does not help. The encoding becomes useful once the model is told how to use it, either through LB\mathcal{L}_B during encoding or through the invocation prompt at inference, and combining both gives the best result.

As for the theory, Appendix B.3 is a proof-of-concept showing that UID conditioning can improve the encoding dynamics, and we do not claim that it explains UID-free question answering. More effective use of UIDs, as well as using UIDs alone, would require the model to be specifically trained to use them, which is an important direction for our future work.

Regarding repeated exposure, all our comparisons are fair, with the same fast weights, optimizer, and stopping criterion for TCE and ACE (Section 4.1). Moreover, Appendix D.2 explicitly matches text exposure. Since each ACE update uses two copies of the context, ACE with KK updates sees the same amount of text as TCE with 2K2K updates. The Recall accuracy (%) of TCE / ACE under matched exposure is reported below (Tables 16–18):

Updates (TCE / ACE)Qwen3.5-4BQwen3.5-9BQwen3.5-27B
8 / 42.0 / 45.70.5 / 34.02.0 / 40.1
16 / 81.0 / 58.92.0 / 44.72.5 / 42.6
32 / 161.0 / 59.42.5 / 50.33.6 / 44.7

As we can see, TCE stays far below ACE under the same text exposure. Together with the UID-free results in the first table, where the UID-conditioned objective alone does not improve over TCE, this shows that the gains are not caused by repeated exposure to the context.

AuthorsOpenReview ↗

Response to Reviewer 2qwe (2/3)

Q2 (W2). The effect of explicit context-use instructions remains unclear. Qwen3.5-9B achieves 94.5% on Counterfactual (Table 2), where queries explicitly instruct the model to use only the context. How much does this depend on the instruction? Would adding the same instruction to Recall improve its 35.5% accuracy (Table 1)? Please compare prompts with and without this instruction while holding the encoded weights and evaluation samples fixed.

A2. We would like to clarify that the instruction is part of our benchmark design, and all methods are given the same instruction, rather than extra information available only to ACE. In Conflict, the context contradicts the model's prior knowledge, so the query has to say which source to use. Without it, the question would simply have no single correct answer. More importantly, the instruction alone does not explain the high accuracy. TCE receives exactly the same Counterfactual queries, but performs far worse than ACE (Table 2, ACE with ci_v1):

CounterfactualQwen3.5-4BQwen3.5-9BQwen3.5-27B
TCE3.514.011.0
ACE92.594.593.0

We have also tried adding an explicit instruction to Recall and Recognition, with the TCE weights and evaluation samples fixed (Appendix D.4). With the generic invocation "Recall the previously encoded context and use information relevant to the following request", the gains are small and inconsistent, and remain far below ACE (Table 20):

LLMMethodRecallRecognition
Qwen3.5-4BTCE6.10.5
TCE + instruction2.00.0
ACE55.381.7
Qwen3.5-9BTCE4.62.0
TCE + instruction9.11.5
ACE35.595.4
Qwen3.5-27BTCE1.50.0
TCE + instruction15.70.0
ACE39.158.4

Therefore, our comparison is fair, and the gains come from how ACE encodes the context, not from a more explicit instruction. For ACE itself, the invocation prompt already explicitly tells the model to treat the encoded chunks as prior context and use the relevant information (Section 3.2), so the effect of such an instruction is already included in ACE's Recall results.

As for why Recall is lower than Counterfactual, Recall is designed to be a harder task, where the model needs to be aware that it has recently acquired new knowledge. Note that the difficulty of the task itself is not a limitation of our method, and including such harder tasks instead reflects the rigor of our evaluation. Moreover, ACE still outperforms the baseline on this hard task, which further demonstrates its effectiveness.

AuthorsOpenReview ↗

Response to Reviewer 2qwe (3/3)

Q3 (W3). Some long-context results require further analysis, and their statistical reliability is unclear. In Table 3, Qwen3.5-9B achieves 3.5 points with ACE on LongBench v2, exceeding the 3.0 points of full-context inference, which is surprising given the very low scores and small difference. Please examine individual examples to explain why ACE performs better. Similarly, the LongBench improvement from 8.2 to 9.6 F1 is reported without variability estimates, confidence intervals, or significance tests. Paired sample-level analysis and uncertainty estimates would help assess whether these gains are reliable.

A3. We would first note that LongBench v2 is the most challenging benchmark in our evaluation. Even full-context inference struggles on it (2.5–4.0 in Table 3), because a question was dropped during its construction if all three reference long-context LLMs could answer it (Bai et al., 2025). Our free-form answer protocol lowers the scores further, since responses that never commit to an option get 0 points. In this hard setting, ACE still matches or improves over TCE on all four models, and reaches scores close to full-context inference.

Following your suggestion, we inspected ten sampled instances on Qwen3.5-9B, among which ACE and full-context inference differ on three:

  • ACE correct, full context incorrect. On a GPT-SoVITS user-manual question, the full-context model ignored the question and wrote "here is a structured summary…" without choosing any option, while ACE chose the gold option (also chosen without the context).
  • ACE partially correct, full context incorrect. On how MagicDrive encodes bounding boxes, the full-context model chose "Fourier embedding for 4 corner point", whereas the document uses 8 corner points. ACE covered part of the gold answer and received partial credit.
  • ACE incorrect, full context correct. On a contract-law question, the full-context model answered correctly, while ACE fell into a repetitive loop ("the offer was not an offer, as …") and was truncated.

So the difference mainly comes from whether the model commits to the correct option under our free-form protocol.

Regarding the LongBench improvement, as you suggested, we further report paired statistics on the 50 instances shared across the four models, including the mean ACE − TCE difference (averaged over the models for each instance), its 95% paired bootstrap confidence interval, the Wilcoxon signed-rank p-value, and the numbers of wins, ties, and losses, under three stopping thresholds τ\tau:

τ\tauACE − TCE (F1)95% CIWilcoxon ppWins / ties / losses
5+1.7[+0.5, +3.0]0.00924 / 10 / 16
8+2.2[+0.6, +4.0]0.01228 / 10 / 12
10+3.1[+1.5, +5.0]<0.00131 / 10 / 9

As we can see, the improvement is statistically significant under all three thresholds.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Encoding context into model weights at test time, which we call TCE, is a promising direction for ultra-long context inference. However, related attempts obtain gains only with meta-training on related domains or prior knowledge of future requests, and neither setting is practical in real applications. We instead study a new and practical setting with only off-the-shelf LLMs and the context itself, thus facing the core difficulty rather than taking a shortcut around it. Our setting is extremely hard, and existing ideas almost completely fail there. To address it, we propose a completely new idea, ACE, which introduces a UID-based addressing mechanism along with a prompting framework, so that the model becomes explicitly controllable in using the encoded information. ACE substantially outperforms the existing methods across models and tasks.

In the following, we summarize the main findings and contributions of this work in three points and explain why each of them matters. Overall, our setting assumes only an off-the-shelf LLM and the context itself, with neither meta-training nor prior knowledge of future requests (Section 2.3). It is more practical yet far more challenging, and to our knowledge, no prior work has really covered it.

  1. We reveal that existing ideas fail once the extra help is removed. With contexts of only about 11 tokens, full-context inference is nearly perfect, while TCE obtains only 1.5–6.1% on Recall and 0.0–2.0% on Recognition (Table 1), and DCD only 5.1–12.2% and 0.0% (A1 in our Response to Reviewer 6Z3e). Thus, the progress of related methods largely relies on meta-training or prior knowledge, which, to our knowledge, has not been examined.
  2. We identify the cause of this failure and propose ACE to solve it. Conventional TCE gives the LLM no cue to access the encoded information and use it as context (Section 3). ACE first introduces a UID that conditions both encoding and inference, explicitly linking the encoded information to later requests. Then, beyond this mechanism, a prompting framework tells the LLM to use this information as context (Sections 3.1–3.2). To our knowledge, this cause has rarely been discussed or addressed, and ACE is a completely new framework for it.
  3. We show that the idea works, with substantial and well-verified gains. ACE substantially improves over TCE, by 35.5–74.1 points on Recall, 91.4–98.0 on Recognition, and 80.0–85.0 on Counterfactual (Tables 1–2). On Qwen3.5-27B, Recall rises from 1.5% to 75.6%, an improvement of more than an order of magnitude (about 50 times). The gains extend to RULER and LongBench (Table 3), and are supported by the ablations (Tables 5, 16–18, 21) and the theory (Appendix B). This shows that ACE is promising and provides a reliable foundation for future work on TCE.

Given the above, we cannot agree with the two extremely negative assessments (Reviewers hVq5 and 2qwe). They do not reflect these findings and contributions, and their main concerns are incorrect, as shown below.

Reviewer hVq5 misunderstands the focus of this paper. We focus primarily on performance, since this central problem for deployment is still far from solved. We consider a very hard setting, where existing methods directly fail due to the lack of the extra information they require. In contrast, ACE is, to our knowledge, the first method that works (Tables 1–2). This makes our paper a zero-to-one work. A dedicated study of capacity and efficiency is thus beyond our scope and a direction for future work, rather than a reason for rejection. Even so, the paper already provides evidence on both, and on the baselines, as shown below.

  1. "The context capacity is unclear." Capacity mainly depends on how many parameters store the context rather than on the encoding algorithm (Appendix C.3), which is beyond our focus. Even so, the paper covers contexts up to 64K tokens (Appendix E.4), with models and tasks comparable to or even broader than related TCE studies.
  2. "The efficiency benefit is unclear." The encoding cost of ACE grows linearly with context length, and on LongBench v2, ACE has lower response latency on all four models and uses about 20–24× fewer FLOPs per query than full-context inference (Appendix F, Tables 33–36).
  3. "The baselines are limited." Existing methods do not fit our setting, and only autoregressive TCE is directly comparable. The most related methods, RECKONING and PERK, use the same encoding as TCE but additionally require meta-training (Section 2.3). We have also added DCD, which stays at the level of TCE and far below ACE (A2 in our Response to Reviewer hVq5).

Reviewer 2qwe overlooks the extensive results already in the paper and selectively criticizes our results intended for stress testing. Stress testing is designed to probe the boundary of ACE after its effectiveness has been verified, which reflects the rigor of our evaluation rather than a weakness. Picking only these results while ignoring the large and consistent gains (Tables 1–2) is not a fair assessment. Moreover, the three analyses requested by the reviewer are already in the paper or in our rebuttal.

  1. "The gains without UID invocation are not explained." This gain is expected under our design, since ACE jointly optimizes its two encoding objectives (Eq. 10), and Tables 4–5 already separate their contributions. It is also not caused by repeated exposure. With the same text exposure, TCE obtains 0.5–3.6% on Recall, while ACE obtains 34.0–59.4% (Tables 16–18).
  2. "The effect of explicit instructions is unclear." The instruction is part of the benchmark design and is the same for all methods, which is completely misinterpreted by the reviewer. With exactly the same queries, TCE obtains 3.5–14.0% on Counterfactual, while ACE obtains 92.5–94.5% (Table 2). Adding an explicit instruction to TCE on Recall and Recognition does not recover the gains either (Table 20).
  3. "The long-context gains may not be reliable." Paired tests on LongBench show that the gains are statistically significant under all three thresholds (A3 in our Response to Reviewer 2qwe). Compared with the baseline, our improvements are also substantial, e.g., from 0–8% to 15–63% on RULER NIAH-3 across the four models, and from 14.0 to 19.2 F1 on LongBench with Muse-Glimmer-30B (Table 3).

In summary, the setting is practical, the method is novel, and the evaluation is extensive. We believe the concerns of the two reviewers do not support a rating of 1, which is confusing to us. We hope that the judgment is based on core problems in our work that are rigorously identified. We sincerely welcome such comments, as an open, objective, and professional discussion is the original purpose of the rebuttal and OpenReview.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.