- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper lets an LM manage its own context by mirroring the live context to an editable file, so that c_{t+1} = f(c_t) instead of append-only. It evaluates this out of the box on deep research, coding and long-horizon tasks, adds in-context learning and RL for context management, and proposes Suffix Cache Reuse (SCR) to cut re-prefilling cost in SGLang.
Critical strengths
- Mapping the context to a real file is an elegant way to realize c_{t+1} = f(c_t). Letting the backend handle synchronization, rather than relying on the model to understand the mechanism, is the right design.
- SCR and the server-side compute measurement in SGLang are impressive and new.
- The paper is concise and not padded.
Critical weaknesses
- No confidence intervals for the main comparisons. Figure 4, Figure 6, Table 1 and Table 2 report point estimates only, and some gaps are small (e.g., 42.5% vs. 42.1% in Table 2; 1.044 vs. 1.026 speedup in Figure 6b). A paired bootstrap is cheap: resample the questions with replacement, recompute each method's score and the difference, repeat ~1,000 times, and report 95% intervals. For EdgeBench-10 and Software World, where tasks are few, variance across independent runs is also needed.
- Out of the box, CLM does not always help weaker models: Qwen3.5-9B scores six points below the summary baseline before RL (Table 2). It would help to discuss how strong a model needs to be for zero-shot CLM to pay off.
Questions for authors
- Qwen3.6-27B is a fairly small model. Without CLM-specific training data, how does it reliably understand the context-as-a-file mapping? Did you construct any data for this?
- Will the full agent framework be open-sourced, e.g., can one run an agent that chats and carries out tasks?
- In some recent frontier models (e.g., Claude Opus 5.5 and Claude Fable 5.1), the reasoning (thinking) is hidden, so CLM cannot edit it through the context file. Several of your agents are API models (Claude 4.6 Sonnet, GPT-5.6-Sol, Opus 5) whose reasoning may likewise not be visible. How is the hidden reasoning handled when the model edits its context? For example, when the model deletes part of its context, the reasoning about that part cannot be deleted with it. Did you test whether this affects performance? The paper does not seem to discuss this; Sections 4.3 and 5.3 mention prior reasoning tokens only in terms of re-prefilling cost. A discussion of this point would be helpful.
4 comments
Q3. In some recent frontier models (e.g., Claude Opus 5.5 and Claude Fable 5.1), the reasoning (thinking) is hidden, so CLM cannot edit it through the context file. Several of your agents are API models (Claude 4.6 Sonnet, GPT-5.6-Sol, Opus 5) whose reasoning may likewise not be visible. How is the hidden reasoning handled when the model edits its context? For example, when the model deletes part of its context, the reasoning about that part cannot be deleted with it. Did you test whether this affects performance? The paper does not seem to discuss this; Sections 4.3 and 5.3 mention prior reasoning tokens only in terms of re-prefilling cost. A discussion of this point would be helpful.
A5. Thank you for raising this great point. When reasoning tokens are hidden in the API model response, we indeed cannot track them in our mirrored context file. The actual impact depends on how the API provider preserves reasoning. We discuss two possible scenarios and provide additional analysis suggesting limited impact in our experiments.
The first possibility is that the API server hides reasoning and removes it from all future turns. This is a common chat endpoint implementation: models such as Qwen and GPT-OSS (released by OpenAI) can omit previous reasoning from subsequent turns in their chat templates, with support from serving engines such as SGLang and vLLM. If the API provider follows this practice, the mismatch between locally maintained conversations and server-side conversations would be limited to reasoning from the latest turn. To examine the impact of reasoning stripping, we evaluated Qwen3.6-27B on BrowseComp-Plus with reasoning stripping enabled and observed no edit failures caused by the stripped reasoning. With prior reasoning stripped or preserved, CLM achieves 59.0% and 58.5%, respectively, on 200 randomly sampled BrowseComp-Plus questions, suggesting limited end-to-end impact in this setting.
The second possibility is that the API server hides reasoning while preserving all previous reasoning content. However, the agent implementations we adopted from Codex and Mini-SWE-Agent maintain conversations locally, prepare new inputs for each request, and rely on automatic prefix cache reuse to save compute. The submitted inputs therefore match the local conversations and exclude previous hidden reasoning. If the server stores that reasoning, restoring it would require additional mechanisms to associate it with the corresponding conversation across incoming requests. We checked the token usage reported by the API and found it consistent with our assumption that each new turn’s server-side inputs align with the locally prepared inputs.
We have added this discussion and the ablation to the revised paper (see Appendix H.8). We welcome further feedback to help us investigate the potential impact of API serving behavior on CLMs.
Thank you for the thorough responses and additional experiments. All of my questions and concerns have been fully addressed, and I will maintain my rating (4: Clear acceptance).
Thank you! We’re glad our response addressed your concerns.
We thank the reviewer for highlighting our contributions to method design and efficiency, as well as the conciseness of our writing. We address the reviewer’s questions below.
W1. No confidence intervals for the main comparisons. Figure 4, Figure 6, Table 1 and Table 2 report point estimates only, and some gaps are small (e.g., 42.5% vs. 42.1% in Table 2; 1.044 vs. 1.026 speedup in Figure 6b). A paired bootstrap is cheap: resample the questions with replacement, recompute each method's score and the difference, repeat ~1,000 times, and report 95% intervals. For EdgeBench-10 and Software World, where tasks are few, variance across independent runs is also needed.
A1. Thank you for this suggestion. We computed paired bootstrap 95% CIs using 2,000 resamples over questions/tasks. The results support our main findings, including CLM’s accuracy gains on BrowseComp-Plus, improvements from RL, and compute savings on EdgeBench-10 and with SCR. For CLM versus Summary after RL (Table 2), which we described as matching performance, the estimated difference is +0.5 points [−4.1, +4.8]. We added the full intervals to the updated paper (see Appendix H.7, Table 10).
For EdgeBench-10, the submission reports the best score over three independent seeds per task (Figure 5a; Appendix F.2). To make variability clearer, we additionally report mean scores across seeds and their confidence intervals here. Under this aggregation, CLM’s score advantage over Summary is +1.7 [−4.6, +7.8], while its compute ratio is 0.41 [0.35, 0.49]. We clarified both aggregation methods and added the paired confidence intervals to Appendix H.7.
Estimating variability is a broader evaluation challenge for expensive, long-horizon benchmarks such as EdgeBench and Software World. EdgeBench’s official documentation estimates hundreds to over a thousand dollars per 12-hour task, while Software World [1]’s official presentation reports individual simulation trajectories without cross-run uncertainty estimates due to prohibitive cost. We added a discussion of these evaluation challenges and our current coverage in the appendix (see Appendix H.7).
[1] Shen, Shannon Zejiang, et al. "World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds." arXiv preprint arXiv:2610.24061 (2026).
W2. Out of the box, CLM does not always help weaker models: Qwen3.5-9B scores six points below the summary baseline before RL (Table 2). It would help to discuss how strong a model needs to be for zero-shot CLM to pay off.
A2. Thank you for this suggestion. We agree that this limitation deserves discussion beyond the RL section. We have updated the out-of-the-box evaluation section (see Section 5.1) to clarify that weaker models may need targeted training to develop reliable context-management skills before CLM consistently improves over summarization.
Q1. Qwen3.6-27B is a fairly small model. Without CLM-specific training data, how does it reliably understand the context-as-a-file mapping? Did you construct any data for this?
A3. We did not construct or use any CLM-specific training data for the out-of-the-box Qwen3.6-27B evaluation. We chose context-as-a-file to build on models’ existing file-editing capabilities: searching files, editing text with shell commands, and moving information between files. The task instructions explain that edits to the designated file update the model’s live context, allowing it to apply familiar operations to context management. We hypothesize that prior training on coding and memory-file management supports this transfer, although we do not know the model’s training data.
Q2. Will the full agent framework be open-sourced, e.g., can one run an agent that chats and carries out tasks?
A4. Yes! We have prepared all materials for open-source release, including our agent framework and scripts for evaluation, in-context learning (ICL), and reinforcement learning (RL). We will link to these materials in the camera-ready version. We have also prepared plug-ins for Pi-Agent so that users can apply CLM to their vibe coding workflows. We will also provide links to additional integrations for Claude Code, Codex, and OpenCode.