acceptodds
Under review as a conference paper at ICLR 2027

Context Language Models

384 views

Abstract

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as an editable file; this naturally extends to multi-agent systems where multiple agent contexts coexist as files. Applying CLMs out of the box outperforms SOTA context-management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 26.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, context management, long-context language models, context editing, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_spL6OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper lets an LM manage its own context by mirroring the live context to an editable file, so that c_{t+1} = f(c_t) instead of append-only. It evaluates this out of the box on deep research, coding and long-horizon tasks, adds in-context learning and RL for context management, and proposes Suffix Cache Reuse (SCR) to cut re-prefilling cost in SGLang.

Critical strengths
  1. Mapping the context to a real file is an elegant way to realize c_{t+1} = f(c_t). Letting the backend handle synchronization, rather than relying on the model to understand the mechanism, is the right design.
  2. SCR and the server-side compute measurement in SGLang are impressive and new.
  3. The paper is concise and not padded.
Critical weaknesses
  1. No confidence intervals for the main comparisons. Figure 4, Figure 6, Table 1 and Table 2 report point estimates only, and some gaps are small (e.g., 42.5% vs. 42.1% in Table 2; 1.044 vs. 1.026 speedup in Figure 6b). A paired bootstrap is cheap: resample the questions with replacement, recompute each method's score and the difference, repeat ~1,000 times, and report 95% intervals. For EdgeBench-10 and Software World, where tasks are few, variance across independent runs is also needed.
  2. Out of the box, CLM does not always help weaker models: Qwen3.5-9B scores six points below the summary baseline before RL (Table 2). It would help to discuss how strong a model needs to be for zero-shot CLM to pay off.
Questions for authors
  1. Qwen3.6-27B is a fairly small model. Without CLM-specific training data, how does it reliably understand the context-as-a-file mapping? Did you construct any data for this?
  2. Will the full agent framework be open-sourced, e.g., can one run an agent that chats and carries out tasks?
  3. In some recent frontier models (e.g., Claude Opus 5.5 and Claude Fable 5.1), the reasoning (thinking) is hidden, so CLM cannot edit it through the context file. Several of your agents are API models (Claude 4.6 Sonnet, GPT-5.6-Sol, Opus 5) whose reasoning may likewise not be visible. How is the hidden reasoning handled when the model edits its context? For example, when the model deletes part of its context, the reasoning about that part cannot be deleted with it. Did you test whether this affects performance? The paper does not seem to discuss this; Sections 4.3 and 5.3 mention prior reasoning tokens only in terms of re-prefilling cost. A discussion of this point would be helpful.
4 comments

We thank the reviewer for highlighting our contributions to method design and efficiency, as well as the conciseness of our writing. We address the reviewer’s questions below.

W1. No confidence intervals for the main comparisons. Figure 4, Figure 6, Table 1 and Table 2 report point estimates only, and some gaps are small (e.g., 42.5% vs. 42.1% in Table 2; 1.044 vs. 1.026 speedup in Figure 6b). A paired bootstrap is cheap: resample the questions with replacement, recompute each method's score and the difference, repeat ~1,000 times, and report 95% intervals. For EdgeBench-10 and Software World, where tasks are few, variance across independent runs is also needed.

A1. Thank you for this suggestion. We computed paired bootstrap 95% CIs using 2,000 resamples over questions/tasks. The results support our main findings, including CLM’s accuracy gains on BrowseComp-Plus, improvements from RL, and compute savings on EdgeBench-10 and with SCR. For CLM versus Summary after RL (Table 2), which we described as matching performance, the estimated difference is +0.5 points [−4.1, +4.8]. We added the full intervals to the updated paper (see Appendix H.7, Table 10).

ComparisonMetricEstimate95% CI
BrowseComp-Plus: CLM − SummaryAccuracy (pp)+5.8[+3.1, +8.4]
TerminalBench 2.1: CLM − SummaryAccuracy (pp)+1.1[−7.9, +9.3]
TBLite: CLM − SummaryAccuracy (pp)+6.7[−2.3, +16.3]
EdgeBench-10: CLM − SummaryScore difference+1.7[−4.6, +7.8]
EdgeBench-10: CLM / SummaryCompute ratio0.41×[0.35, 0.49]
EdgeBench-10: CLM − BaseScore difference+22.1[+7.9, +38.3]
Table 2: CLM after − before RLAccuracy (pp)+13.7[+9.3, +18.4]
Table 2: CLM − Summary after RLAccuracy (pp)+0.5[−4.1, +4.8]
SCR: SCR − StandardAccuracy (pp)+1.6[+0.1, +3.1]
SCR: SCR / StandardFLOPs ratio0.655×[0.642, 0.669]

For EdgeBench-10, the submission reports the best score over three independent seeds per task (Figure 5a; Appendix F.2). To make variability clearer, we additionally report mean scores across seeds and their confidence intervals here. Under this aggregation, CLM’s score advantage over Summary is +1.7 [−4.6, +7.8], while its compute ratio is 0.41 [0.35, 0.49]. We clarified both aggregation methods and added the paired confidence intervals to Appendix H.7.

Estimating variability is a broader evaluation challenge for expensive, long-horizon benchmarks such as EdgeBench and Software World. EdgeBench’s official documentation estimates hundreds to over a thousand dollars per 12-hour task, while Software World [1]’s official presentation reports individual simulation trajectories without cross-run uncertainty estimates due to prohibitive cost. We added a discussion of these evaluation challenges and our current coverage in the appendix (see Appendix H.7).

[1] Shen, Shannon Zejiang, et al. "World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds." arXiv preprint arXiv:2610.24061 (2026).

W2. Out of the box, CLM does not always help weaker models: Qwen3.5-9B scores six points below the summary baseline before RL (Table 2). It would help to discuss how strong a model needs to be for zero-shot CLM to pay off.

A2. Thank you for this suggestion. We agree that this limitation deserves discussion beyond the RL section. We have updated the out-of-the-box evaluation section (see Section 5.1) to clarify that weaker models may need targeted training to develop reliable context-management skills before CLM consistently improves over summarization.

Q1. Qwen3.6-27B is a fairly small model. Without CLM-specific training data, how does it reliably understand the context-as-a-file mapping? Did you construct any data for this?

A3. We did not construct or use any CLM-specific training data for the out-of-the-box Qwen3.6-27B evaluation. We chose context-as-a-file to build on models’ existing file-editing capabilities: searching files, editing text with shell commands, and moving information between files. The task instructions explain that edits to the designated file update the model’s live context, allowing it to apply familiar operations to context management. We hypothesize that prior training on coding and memory-file management supports this transfer, although we do not know the model’s training data.

Q2. Will the full agent framework be open-sourced, e.g., can one run an agent that chats and carries out tasks?

A4. Yes! We have prepared all materials for open-source release, including our agent framework and scripts for evaluation, in-context learning (ICL), and reinforcement learning (RL). We will link to these materials in the camera-ready version. We have also prepared plug-ins for Pi-Agent so that users can apply CLM to their vibe coding workflows. We will also provide links to additional integrations for Claude Code, Codex, and OpenCode.

Q3. In some recent frontier models (e.g., Claude Opus 5.5 and Claude Fable 5.1), the reasoning (thinking) is hidden, so CLM cannot edit it through the context file. Several of your agents are API models (Claude 4.6 Sonnet, GPT-5.6-Sol, Opus 5) whose reasoning may likewise not be visible. How is the hidden reasoning handled when the model edits its context? For example, when the model deletes part of its context, the reasoning about that part cannot be deleted with it. Did you test whether this affects performance? The paper does not seem to discuss this; Sections 4.3 and 5.3 mention prior reasoning tokens only in terms of re-prefilling cost. A discussion of this point would be helpful.

A5. Thank you for raising this great point. When reasoning tokens are hidden in the API model response, we indeed cannot track them in our mirrored context file. The actual impact depends on how the API provider preserves reasoning. We discuss two possible scenarios and provide additional analysis suggesting limited impact in our experiments.

The first possibility is that the API server hides reasoning and removes it from all future turns. This is a common chat endpoint implementation: models such as Qwen and GPT-OSS (released by OpenAI) can omit previous reasoning from subsequent turns in their chat templates, with support from serving engines such as SGLang and vLLM. If the API provider follows this practice, the mismatch between locally maintained conversations and server-side conversations would be limited to reasoning from the latest turn. To examine the impact of reasoning stripping, we evaluated Qwen3.6-27B on BrowseComp-Plus with reasoning stripping enabled and observed no edit failures caused by the stripped reasoning. With prior reasoning stripped or preserved, CLM achieves 59.0% and 58.5%, respectively, on 200 randomly sampled BrowseComp-Plus questions, suggesting limited end-to-end impact in this setting.

The second possibility is that the API server hides reasoning while preserving all previous reasoning content. However, the agent implementations we adopted from Codex and Mini-SWE-Agent maintain conversations locally, prepare new inputs for each request, and rely on automatic prefix cache reuse to save compute. The submitted inputs therefore match the local conversations and exclude previous hidden reasoning. If the server stores that reasoning, restoring it would require additional mechanisms to associate it with the corresponding conversation across incoming requests. We checked the token usage reported by the API and found it consistent with our assumption that each new turn’s server-side inputs align with the locally prepared inputs.

We have added this discussion and the ablation to the revised paper (see Appendix H.8). We welcome further feedback to help us investigate the potential impact of API serving behavior on CLMs.

Thank you for the thorough responses and additional experiments. All of my questions and concerns have been fully addressed, and I will maintain my rating (4: Clear acceptance).

Thank you! We’re glad our response addressed your concerns.

Reviewer_ESWeOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Context Language Models (CLMs), which allow models to directly manage the context via an editable file. Unlike existing methods that rely on predefined compression or retrieval operations, CLMs can freely modify, reorganize, and maintain their context. The paper further explores improving context management through in-context skill evolution and reinforcement learning, and proposes suffix cache reuse to reduce re-prefilling costs after context edits.

Critical strengths
  1. CLMs propose a novel context management design that allows models to directly edit their own context, enabling more flexible and task-relevant management strategies.
  2. CLMs is simple and plug-and-play: it reuses existing file operation capabilities, requires no new tool interfaces for context management strategies, and easily extends to multi-agent scenarios.
  3. Evaluations on Skill Evolution and RL show that CLMs' context management capability can be further optimized.
  4. The method mitigates the re-prefill overhead through suffix cache reuse, improving its practicality.
Critical weaknesses
  1. Lack of fine-grained analysis: The paper only reports final performance; lacks systematically analysis when agents edit, how much each edit helps, how often edits lose important information.
  2. It unclear how much of the RL improvement comes from better context editing rather than general task learning.
  3. Efficiency is primarily evaluated on prefix-reuse FLOPs. A more complete analysis of memory usage, and context-editing overhead would help a lot.
Questions for authors
  • How much of CLM's improvement comes from unrestricted context editing? Can you provide a case study of final performance with and without editing operations?
  • How much does suffix cache reuse depend on where and how much the context is edited? Are there cases where reusing old KV states leads to performance loss?
  • CLMs does not fully discuss the latency overhead of autonomous context editing. Compared with fixed context-management strategies, how does CLM's end-to-end latency perform, after including the overhead of generating and executing edits?
7 comments

We thank the reviewer for acknowledging our contributions and raising insightful questions. We address the questions below.

W1. Lack of fine-grained analysis: The paper only reports final performance; lacks systematically analysis when agents edit, how much each edit helps, how often edits lose important information.

A1. Thank you for this suggestion. We agree that fine-grained analysis can help us better understand and improve CLM behavior. We further examined edit motivations, information loss, and ways to score each edit’s helpfulness. Our analysis suggests that agents edit both to manage context pressure and to organize task information, with the balance varying across tasks. Sampled edits generally preserve information needed later, though omissions and factual distortions occasionally occur. We also explore scoring intermediate edits using LLM-as-a-judge and Monte Carlo sampling. We detail these findings below.

When does the agent edit? Creating a comprehensive categorization of editing behavior is challenging, as we allow LMs to perform arbitrary manipulations of their context. We use LLM-as-a-judge to perform a preliminary analysis of edit timing and intention and have added a discussion of this analysis to the updated discussion section (see Appendix H.2 for details). We first prompted the LLM judge to classify whether each edit was motivated by context pressure, with the model explicitly stating that it needed to compact the context to continue working, or by task organization under low context pressure. We found that 38% of edits on BrowseComp-Plus and 67.5% on TerminalBench 2.1 were motivated by context pressure, with the remaining edits tagged as intended for task organization. In addition, we asked the LLM judge to tag when and where the edits occur on BrowseComp-Plus and cluster them to identify the three most common patterns: 48% follow a period without progress, 36% follow a resolved sub-question, and 24% follow a large tool output (categories can overlap for the same edit). We also note that model behaviors are highly task-dependent. For example, as shown in Figure 9 of our submission, the agent performs more creative edits when it faces more complex context management needs, such as multi-agent orchestration and progress tracking.

How much does editing help or lose important information? Our submission’s end-to-end evaluations show that CLMs outperform Summary by 6.1–6.7 percentage points and Self-Compact by 14.0–23.7 points on BCP and TBLite, while matching Summary on TB2.1 with 29.5% less compute (Figure 4). Evaluating the contribution of intermediate actions has been a long-standing challenge in process reward design. Each edit affects subsequent actions, making its individual contribution to final performance and efficiency difficult to measure. As a preliminary study, we explore two approaches to this analysis: LLM-as-a-judge and Monte Carlo sampling.

We first use an LLM to assess edit helpfulness and information preservation under a fixed rubric (see Appendix H.9 of the updated revision). Among 100 randomly sampled BrowseComp-Plus edits, 97% are classified as “helpful,” 2% as “neutral,” and 1% as “harmful.” The rubric considers whether edits free useful context budget while preserving needed information, and flags information loss with visible downstream consequences or factual distortion as harmful. These judgments provide a preliminary assessment of edit quality, but quantifying how much each edit contributes remains challenging with a single LLM call.

Another common approach to estimating process rewards is Monte Carlo sampling. We explored this approach by continuing generation with and without each edit six times each and comparing the average final scores. We keep subsequent editing available in both conditions, preserving the model’s ability to edit throughout the rollout. Consistent with this challenge, applying the edit changed final performance by an average of +2.0 pp, with a median of 0 and no difference in 62% of cases. These results highlight the difficulty of using rollout-level outcomes to assess individual context edits: Applying it to context editing is challenging because later edits can compensate for a skipped edit, making its individual contribution difficult to capture through final performance alone.

Overall, we find it challenging to evaluate individual context editing behavior without accounting for the compounding effects of edits within each trajectory. Therefore, we leave the development of more effective methods for fine-grained context management analysis to future work. We have included this analysis in the revised paper (Appendices H.2 and H.3) and would welcome further feedback from the reviewer to guide additional analysis.

W2. It unclear how much of the RL improvement comes from better context editing rather than general task learning.

A2. Thank you for raising this. In Table 2 of the submission, we ran an ablation to control for general task learning so that we can isolate how much of the improvement comes from better context editing compared with a hard-coded compaction operation when it reaches the limit. We found both reach similar final accuracy (42.1% for Summary and 42.5% for CLM), while CLM uses only 61% of Summary’s prefix-reuse FLOPs. Both models learned to manage context to retain key information and keep searching until they found the answer. The 9B CLM performed poorly before training, then caught up to the trained baseline while using less compute. One possible reason CLM matches Summary in accuracy after training is that BrowseComp-Plus mainly requires sustained search and retention of key information, which may require little flexibility in context management.

Due to resource limitation, our RL experiments remained small-scale. We believe this question deserves further study. Scaling up RL may reveal additional context-management behaviors, and tasks with more complex context requirements may offer greater opportunities for learned editing to improve accuracy as well as efficiency. We included a discussion on these possibilities in the revision (see Appendix H.4).

W3. Efficiency is primarily evaluated on prefix-reuse FLOPs. A more complete analysis of memory usage, and context-editing overhead would help a lot.

A3 (W3). To measure memory usage and context-editing overhead, we ran an additional comparison of CLM and the summary harness, the second-best method by accuracy on BrowseComp-Plus. We evaluated both methods on the same 200 questions using Qwen3.6-27B served with SGLang on a single H200. Both runs used concurrency 8 and the same KV-pool size, and both retained reasoning across turns.

MetricCLMSummary harness
Peak GPU memory (weights + KV pool)107 GB107 GB
Peak KV-pool occupancy34%34%
Mean end-to-end latency per question394 s421 s
Context-management share of wall time21%26%
Mean server-forwarded tokens per question133K154K
Mean server FLOPs per question (prefill + decode)7.1 PFLOPs8.2 PFLOPs
Accuracy58.5%53.5%

Peak GPU memory and peak KV-pool occupancy were identical across the two runs. Most of CLM’s editing cost came from generating edit commands. Applying edits took 0.2 s per question, and synchronizing the context file across turns took 2.8 s per question. Together, these operations accounted for less than 1% of CLM’s end-to-end latency. Therefore, CLMs were still faster. We have added these measurements and the latency breakdown to the revised paper (see Appendix H.5, Table 7).

Q1. How much of CLM's improvement comes from unrestricted context editing? Can you provide a case study of final performance with and without editing operations?

A4. Thank you for this question. On BrowseComp-Plus, the same model scores 59.4% with CLM and 26.5% without context management, a gain of 32.9 percentage points (Figure 4a). To assess the benefit of greater editing flexibility, we also compared CLM with three baselines. The summary harness asks the model to summarize when context usage reaches 75%. Self-Compact [1] lets the model decide when to compact, and ACM [2] provides predefined operations for offloading and retrieval. CLM scores 6.1–43.4 percentage points higher than these three baselines.

Our original submission includes two examples of how models use editing flexibility. ContextBench illustrates how editing flexibility affects performance on tasks designed to test specific context-management skills (Figure 2). On Sudoku Sketchpad, for example, CLM can update individual cells as moves arrive, while methods without in-place editing must regenerate the board. In addition, Figure 9 shows how the model uses this flexibility on BrowseComp-Plus: it wrote a helper function and reused it 37 times to compact old search results while preserving a progress note. We’d be happy to clarify either example or address any further questions.

Case study with and without editing. On one BrowseComp-Plus question, the agent without editing found three of the four supporting documents, then ran out of context on another search without answering. With editing enabled, the agent rewrote its notes at step 24, reducing its context from 24.6K to 11.7K tokens while preserving the supporting document IDs and its candidate answer. It answered correctly eight steps later.

Case study of unrestricted editing versus summarization. On another question, CLM updated the same note across six edits to record which candidates it had ruled out and ultimately answered correctly. The summary harness lost track of an earlier elimination and returned a candidate it had already ruled out.

Our view is that expanding the action space can raise the ceiling on what agents can achieve by giving them more freedom to act beyond the policies predefined by an external harness. Our learning algorithm could further help agents explore this richer space and learn more effective strategies. We have updated the submission with the above analysis and case study examples in Appendix H.3.

[1] Li, Tianjian, et al. "Self-Compacting Language Model Agents." arXiv preprint arXiv:2606.23525 (2026).

[2] Li, Xiaochuan, et al. "Acm: Agentic context management for long horizon tasks." arXiv preprint arXiv:2607.23809 (2026).

Q2. How much does suffix cache reuse depend on where and how much the context is edited? Are there cases where reusing old KV states leads to performance loss?

A5. Thank you for this question. We conducted two new experiments: a controlled sweep of edit locations and sizes on real BrowseComp-Plus contexts, and a targeted stress test designed to expose failures from stale cached information.

(continued in the next comment.)

Reviewer_4QfNOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Context Language Models (CLMs), language models that natively manage their own context instead of relying on external harnesses. The key idea is to treat the live context as an editable file: the model can append tokens as usual or freely rewrite its context via Bash commands, with edits synchronized to the live context each turn. The authors further propose ContextBench, a diagnostic benchmark isolating context-management skill; prefix-reuse FLOPs, an inference-cost metric accounting for re-prefilling after edits; Suffix Cache Reuse for efficient serving of edited contexts; and learning methods including natural-language steering, textual skill evolution, and stepwise GRPO with a success-gated efficiency advantage. Experiments across deep research, terminal coding, mathematical optimization, and 24-hour agent swarms show CLMs outperforming strong baselines while using substantially less compute.

Critical strengths

The idea is well-motivated. Eq. (2) subsumes prior tool-based context management as a special case and pushes model autonomy to its logical limit, echoing the Bitter Lesson.

The authors pair a diagnostic benchmark (ContextBench) that isolates context management from reasoning with frontier long-horizon benchmarks, demonstrating consistent accuracy gains at lower compute.

Steering, skill evolution, and RL show context management is a searchable, learnable capability; the success-gated efficiency advantage is a thoughtful reward design. Qualitative examples (trackers, new roles, reusable functions) are compelling.

Critical weaknesses

I do not identify critical weaknesses in the paper.

Questions for authors
  • You discuss in Future Direction that one may distill existing harnesses into CLMs (harness for CLM). The other way around, how do you envision CLM could benefit or improve current harness distillation methods (e.g., Harness-Zero [1])?
  • Most experiments cap the model context budget at 32K tokens. What is the effect of scaling the context budget further (e.g., to 128K or 1M)? As context pressure diminishes, would the gap between CLM and harness-based baselines shrink?
  • Do you expect self-context-editing to become a capability that frontier labs natively support through released harnesses and model training?

[1] Ye, H., Lu, Y., Dong, H., Su, Z., & Song, G. (2026). Harness-Zero: Harness Distillation via Agent-as-Harness. arXiv preprint arXiv:2609.24974.

3 comments

We thank the reviewer for highlighting our contributions and asking inspiring questions for future work. Please find our answers to these questions below:

Q1. You discuss in Future Direction that one may distill existing harnesses into CLMs (harness for CLM). The other way around, how do you envision CLM could benefit or improve current harness distillation methods (e.g., Harness-Zero [1])?

A1. Thank you for this insightful question. Harness-Zero and CLMs share the goal of moving harness-controlled behavior into model policies. This requires the target harness’s action space to be sufficiently flexible to capture the source harness’ behavior. A CLM is helpful as a target harness here because it accommodates all sorts of source harness behaviors: Any transformation the source harness applies to the model’s input can be expressed as a context edit, allowing the student to learn when and how to make that transformation itself. In contrast, a target harness without context-editing in the action space can only internalize behaviors that only append new information in the context. We have added this discussion to the discussion section in the revision (see Appendix H.4).

Q2. Most experiments cap the model context budget at 32K tokens. What is the effect of scaling the context budget further (e.g., to 128K or 1M)? As context pressure diminishes, would the gap between CLM and harness-based baselines shrink?

A2. Thank you for raising this question. To address it, we first conducted additional experiments on all 830 BrowseComp-Plus questions with Qwen3.6-27B at 32K, 64K, and 128K, respectively. We report paired bootstrap 95% CIs for the improvements (2,000 resamples over questions).

BudgetCLMSummaryCLM − Summary (95% CI, pp)UnmanagedUnmanaged overflow
32K59.152.8+6.3 [+3.3, +9.5]21.177.8%
64K61.656.6+5.0 [+2.0, +7.9]46.550.5%
128K60.152.9+7.2 [+4.0, +10.5]56.68.7%

In the table above, we show that the gap does not shrink over the tested range: CLM maintains a statistically significant 5.0–7.2-point advantage over Summary. The two rightmost columns report accuracy and overflow rates with context management disabled. On BrowseComp-Plus, only 8.7% of rollouts overflow at 128K, compared with 77.8% at 32K. We chose 32K because it allows us to study frequent context management decisions in a controlled setting for these relatively shorter-horizon tasks.

For longer-horizon tasks, we also included a 272K-context evaluation on SoftwareWorld in our original submission. This evaluation ran for over 1 day and required substantial context management even with a 272K context window. In the revised manuscript, we added an EdgeBench evaluation with a 128K context window to Appendix H.1, Table 6. At 128K, CLM uses 0.64× the compute of Summary, with comparable scores (47.3 vs. 47.8). Simply allowing CLM to create additional context files enables subagents and raises its score to 50.2, while still using slightly less compute than Summary (219 vs. 222 PFLOPs). These results suggest that lower context pressure can shift the benefit toward efficiency. We did not test 1M because it exceeds the supported context length of our open-model setup and would likely introduce little context pressure on many existing benchmark tasks. Meaningfully studying that regime requires longer tasks and substantially greater evaluation resources. We have included these new results and discussions in the revised manuscript (see Appendix H.1, Tables 5 and 6).

Q3. Do you expect self-context-editing to become a capability that frontier labs natively support through released harnesses and model training?

A3. Yes! We hope so, because our paper shows that this already works well out of the box, can be made even better with training at small scales, and we believe that it'll be even more effective with larger models that can find more creative ways of using context. For frontier labs, serving efficiency is a big deal so the reduction in PFLOPs by CLMs, even without suffix cache reuse, could represent significant savings. But at the same time, the infra stacks might have to better support CLMs: for example, they could strengthen safety protocols to prevent models from removing important safety constraints, make context edits reversible and interpretable (so that failures can be traced), and support self-context editing during training.

Reviewer_4QfNOpenReview ↗

Thanks for your response

Thank you for your response, and my concerns have been addressed. This is highly insightful and pioneering work, and I enjoyed reading it a lot.

Thank you for your kind words! We’re glad our response addressed your concerns!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.