- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAP for managing the history of multi-turn coding agents. It retains the full conversation archive while maintaining a lightweight graph of message blocks. The graph records attention-derived importance scores and links between blocks; when a new user request arrives, the method combines these signals with identifier overlap to select a smaller working context without another model call.The problem this paper addresses is real. It proposes a memory compression approach based on blocks and a block index. In essence, it stores the memory as blocks elsewhere, retains a summary, and uses heuristic retrieval to bring relevant blocks back. However, I see several problems. It does not actually reduce total storage requirements; it may simply move the memory from RAM to disk, while still requiring frequent transfers between them. The memory selection process relies heavily on heuristics and empirically tuned parameters. The paper offers no substantive theory explaining the relationship between attention and importance, or why its various thresholds and weights were chosen. The experimental evaluation is incomplete: it does not compare against some of the methods discussed in the related work, and the results are not sufficiently convincing. Overall, I would recommend rejection.
Critical strengths
The paper addresses a real problem: long agent histories increase context costs, while future user requests may need details that were previously omitted. The method can retrieve such details from the archive. Its Lost-in-Conversation results are substantial, and the reported context-selection cost is low. The evaluation includes full history, recency, random selection, compression methods, KV eviction, and a Codex-style summarization baseline.
Critical weaknesses
My main concern is that the evidence does not yet isolate why the proposed selection rule works or establish its full operational cost. Attention is used both as an importance signal and as a proxy for dependency, but the paper does not establish that an attention link reliably identifies context needed for a later task. The choices of thresholds and weights receive limited justification or sensitivity analysis. Moreover, the “no attention graph” ablation removes importance scores, dependency links, and stale-block handling together, so it cannot attribute the observed gain to any one component.
The efficiency claim also needs a clearer accounting. RECAP reduces the working context, but it retains the complete text archive; the reported graph-memory figure excludes that archive. The paper does not measure archive storage or retrieval I/O, nor does it provide an end-to-end measurement covering the overhead of collecting attention statistics. Its cold-restoration latency result combines separately measured compression time with synthetic cold-request TTFT, rather than measuring the complete workflow.
Finally, the quality–context trade-off is mixed. On SWE-Together with Qwen3-Coder, the Codex-style summary attains a slightly higher reported score than RECAP (28.7 versus 28.5) while using substantially fewer context tokens (5,239 versus 15,344). The Lost-in-Conversation gains are compelling, but matched-budget comparisons and more targeted ablations would make the overall conclusion more convincing.
Questions for authors
How sensitive are the results to the attention threshold and the importance/query weights? What are the separate contributions of node importance, dependency expansion, stale-block handling, and protected records? What are the end-to-end costs of attention collection and archive retrieval? Can the authors report quality across matched context budgets, particularly against the summarization baseline?
8 comments
Response to Reviewer uPXN (Part 2)
[W1&Q1] Sensitivity to the attention threshold and to the importance and query weights.
We vary the query weight λ, the importance update rate α, and the dependency expansion allowance δ around their shared defaults. These parameters control how the method balances current relevance, historical attention, and supporting context.
- We change λ from 0.5 to 0.25 and 1.0 on LiC with gpt-oss and SWE-Together with Qwen3-Coder.
- We change the importance EMA rate α from 0.6 to 1.0. With α = 1.0, only the latest observed attention contributes to the updated score.
- We increase the one-hop expansion allowance δ from 0.1 to 0.2, alongside the separate no-expansion ablation in Table 1. This tests the influence of the expansion budget. The edge threshold θ determines which links are formed and is not varied in this experiment.
The results are as follows:
[Table 3] Parameter sensitivity with shared defaults λ = 0.5, α = 0.6, δ = 0.1
| Variant | Benchmark | Default | Variant | Δ |
|---|---|---|---|---|
| λ = 0.25 | LiC | 67.6 | 67.0 | +0.6 |
| λ = 0.25 | SWE-Together | 28.5 | 29.4 | -0.9 |
| λ = 1.0 | LiC | 67.6 | 66.8 | +0.8 |
| λ = 1.0 | SWE-Together | 28.5 | 27.7 | +0.8 |
| α = 1.0 | LiC | 67.6 | 64.4 | +3.2 |
| α = 1.0 | SWE-Together | 28.5 | 29.9 | -1.4 |
| δ = 0.2 | SWE-Together | 28.5 | 30.3 | -1.8 |
We define Δ as the default score minus the variant score. No tested alternative improves both benchmarks consistently. For example, λ = 0.25 and α = 1.0 improve SWE-Together reward but reduce LiC accuracy. Increasing δ to 0.2 raises SWE-Together reward from 28.5 to 30.3. The results show that shared defaults work across tasks, while a larger expansion budget can help on SWE-Together. These tests do not establish sensitivity to θ, which would require a direct threshold sweep.
[W2&Q3] Archive storage, retrieval I/O, and the end-to-end cost of attention collection.
The efficiency question concerns the complete system, not just cheap selection. We therefore measure archive storage and retrieval, the cost of obtaining attention, and request-path latency over real sessions. These tests show where the savings come from and which operations add overhead.
- We measure the size of the text archive, the time to read a recorded request, and the time to assemble selected context on SWE-Together (Table 4).
- We measure additional native attention collection cost and the latency of a separate 0.6B proxy pass at different context lengths (Table 5).
- We replay 30 real SWE-Together sessions with 280 user boundaries, measuring first-token latency, compaction, and post-response graph work (Table 6).
The results are as follows:
[Table 4] SWE-Together archive storage and retrieval cost
| Cost | |
|---|---|
| Text archive per session, median | 549 KB (1,265 KB) |
| KV cache for the same history, median | 12.2 GiB (28.0 GiB) |
| Read one recorded request | 0.80 ms |
| Assemble the selected context | 0.13 ms |
[Table 5] Cost of collecting attention per request (H100, medians of 10 runs)
| Context tokens | 1K | 4K | 16K | 32K | 65K |
|---|---|---|---|---|---|
| Prefill (Qwen3-Coder-30B) | 0.068 s | 0.182 s | 0.810 s | 2.078 s | 6.215 s |
| + Attention collection | +0.012 s | +0.034 s | +0.199 s | +0.463 s | +1.441 s |
| Extra GPU memory | 0.02 GiB | 0.29 GiB | 1.31 GiB | 2.58 GiB | 5.19 GiB |
| 0.6B proxy pass instead | — | 0.059 s | 0.315 s | 0.861 s | 2.656 s |
[Table 6] Complete workflow on 30 SWE-Together sessions. Medians over 280 user boundaries
| Full history, cold | Full history, prefix cache | Codex-style summary | ReCAP | |
|---|---|---|---|---|
| First-token latency | 9.92 s | 0.95 s | 10.78 s | 0.93 s |
| Compaction time (included above) | 0 | 0 | 7.57 s | 0.016 s |
| Prompt tokens | 88,384 | 88,384 | 13,167 | 19,920 |
| Graph update after the response | — | — | — | 1.20 s |
Response to Reviewer uPXN (Part 3)
These measurements separate persistent storage, signal collection, and the benefit at a cache miss.
- The text archive occupies a median 549 KB per SWE-Together session, compared with 12.2 GiB of KV cache for the same history. This archive is the transcript the agent harness already keeps, rather than an additional KV state. Reading a recorded request takes 0.80 ms, and assembling the selected context takes 0.13 ms (Table 4).
- Native attention collection adds 18% to 25% to a same-length Qwen3-Coder prefill. A separate 0.6B proxy pass takes 0.059 to 2.656 s for contexts from 4K to 65K tokens (Table 5). These measurements distinguish the cost of obtaining attention from CPU-based context selection.
- The session replay measures first-token latency of 0.93 s for RECAP, 9.92 s for cold full history, and 10.78 s for Codex summarization. Warm full history takes 0.95 s with a prefix cache (Table 6). RECAP selection takes 0.016 s, while 1.20 s of post-response graph work is reported separately. These direct measurements replace the paper's earlier synthetic cold estimate.
[W3&Q4] The quality–context trade-off is mixed; quality across matched context budgets, particularly against the summarization baseline.
We add two comparisons to test whether the quality advantage persists under tighter context budgets. The first compares RECAP with a budget-bounded summarizer at the same nominal retention fractions on LiC. The second tightens RECAP's budget on SWE-Together until its realized context is shorter than the Codex-style summary.
- On LiC, both methods use β values of 0.05, 0.10, and 0.20. This holds the nominal budget setting fixed while testing task quality across a range of retention levels (Table 7).
- On SWE-Together, we test a tighter RECAP configuration that retains 4,809 tokens per call, below the 5,239 used by Codex-style summarization (Table 8). This directly addresses the reviewer's concern about the higher context use of RECAP at its original default setting.
The results are as follows:
[Table 7] LiC accuracy for RECAP and budget-bounded summarization at shared nominal retention fractions
| Backbone | β | ReCAP | Summary | Δ |
|---|---|---|---|---|
| gpt-oss | 0.05 | 70.4 | 61.2 | +9.2 |
| gpt-oss | 0.10 | 67.6 | 57.2 | +10.4 |
| gpt-oss | 0.20 | 55.2 | 46.6 | +8.6 |
| Qwen3-Coder | 0.05 | 86.4 | 84.8 | +1.6 |
| Qwen3-Coder | 0.10 | 85.4 | 81.8 | +3.6 |
| Qwen3-Coder | 0.20 | 81.2 | 80.6 | +0.6 |
[Table 8] SWE-Together reward, realized context, and first-token latency with tighter RECAP retention
| Reward | Context tokens | First-token latency | |
|---|---|---|---|
| ReCAP | 31.9 | 4,809 | 0.44 s |
| Codex-style summary | 28.7 | 5,239 | 7.78 s |
Together, the comparisons show that RECAP retains a quality advantage across nominal LiC budgets and under a tighter SWE-Together context cap.
- RECAP outperforms budget-bounded summarization at all three nominal retention fractions on both LiC backbones. Its gains are 9.2, 10.4, and 8.6 accuracy points with gpt-oss, and 1.6, 3.6, and 0.6 points with Qwen3-Coder (Table 7).
- On SWE-Together, the tighter RECAP setting reaches 31.9 reward with 4,809 context tokens, compared with 28.7 and 5,239 for Codex-style summarization. It also lowers measured first-token latency from 7.78 s to 0.44 s (Table 8). RECAP therefore performs better while using fewer tokens in this controlled setting.
- In the paper's original default-budget comparison, summarization scored 28.7 against RECAP's 28.5 and used fewer context tokens. That comparison did not match realized context length. The new tighter-budget result addresses this directly. RECAP also avoids a summarization model call and can restore history that was omitted at an earlier turn.
Response to Reviewer uPXN (Part 1)
We thank the reviewer for the detailed assessment and for recognizing the importance of efficient agent memory, the strong results, and the low cost of RECAP. We address the mechanism, system costs, and quality comparisons with additional experiments.
[W1&Q2] The evidence does not isolate why the selection rule works: separate contributions of node importance, dependency expansion, stale-block handling and protected records, and whether an attention link identifies context needed later.
The earlier no-graph ablation removed multiple components together. We now separate their contributions and test two properties of attention links. The component ablations measure task quality. Next-turn attention tests whether the ranking predicts later reuse. Parent removal tests whether linked supporting blocks matter after selection.
The isolated ablations retain all components except the one removed. We report SWE-Together reward with Qwen3-Coder-30B-A3B and LiC accuracy with gpt-oss-20b, both scaled by 100. Each Δ is the score with the component minus the score without it. The linked-parent removal test also includes a setting without mandatory protected records.
The results are as follows:
[Table 1] Isolated contributions of RECAP components on SWE-Together and LiC
Stale handling is not triggered by the LiC histories in these runs, so we report this ablation only on SWE-Together.
[Table 2] Removing linked parents versus random selected blocks on LiC without mandatory protection
The results show complementary contributions from persistent attention and dependency links.