- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CONTEXTRENDER, a context manager for tool-using LLM agents. It stores tool outputs in a persistent graph and selects which results to include under a fixed history budget. Selection combines recency and semantic relevance with observed reuse, which tracks when later execution steps reuse identifiers from earlier tool results. The evaluation covers AppWorld and an eight-objective QA benchmark with three execution models. At a 6K history budget, the reported results improve over the evaluated context-management baselines and approach or exceed full-history performance at lower estimated cost. The reuse ablation suggests that this signal is particularly helpful for retaining useful results from much earlier in the trajectory. CONTEXTRENDER FROM EXECUTION DE…
Critical strengths
I like the use of observed reuse as a signal for context selection. An older result can still matter even when it does not look especially relevant to the current step. Tracking its later use gives a concrete reason to retain it. The matching rules and selection policy are explicit, so the approach is relatively easy to inspect and implement. This makes it a useful contribution to a practical problem in long-running agents.
The reuse ablation is the most convincing part of the evaluation for me. It connects the proposed signal to both task performance and Later-Used Result Recall, particularly at long lookback distances. The experiments across two benchmarks and three models also gives some evidence that the benefit is not specific to one agent setup. I appreciate that the evaluation includes a measure of what the context manager actually retains, alongside final task scores.
Critical weaknesses
My main concern is how reliably the experiments establish the reported gains. The main results use one run per task without confidence intervals, which makes the smaller differences difficult to interpret. The Semantic Retrieval baseline also reserves 60% of its budget for recent turns, and the cost estimates apply the Full-history cache rate to all methods. Neither choice is necessarily unreasonable, but the conclusions may depend on them. I would be more convinced by uncertainty estimates and a sensitivity analysis showing whether the performance and cost advantages survive changes to these assumptions.
The coverage of the reuse signal is also unclear. Exact, case-sensitive matching can miss paraphrases and transformed values, while the first tool result containing a value may not be its actual source. A single reused value also increases the priority of the whole result, potentially bringing back irrelevant content. These seem like reasonable compromises for a lightweight heuristic, but I would like to know how often they matter in the experiments. Without coverage statistics or a breakdown of failure cases, it is hard to judge how well the method would work when tool outputs contain mostly free-form text rather than stable identifiers.
Questions for authors
What fraction of tool results receives a non-zero reuse score? Could you report this seperately for each benchmark and include a few examples where exact matching misses useful context or attributes reuse to the wrong result? This would help clarify how much of the observed benefit depends on identifier-heavy environments.
Could you report confidence intervals for the main performance differences and repeat the evaluation on a representative subset of tasks? I am particularly interested in whether the smaller gains remain consistent across repeated agent runs.
How sensitive are the comparisons to the Semantic Retrieval baseline’s fixed 60% allocation? Also, do the estimated cost savings remain similar when each method’s own cache behavior is used instead of the Full-history cache rate? These checks would help assess the practical advantage claimed for the method.
3 comments
Q2. Uncertainty across tasks and repeated executions.
The paired bootstrap intervals support positive QA F1 gains against Recency + Summary, Semantic Retrieval, and ACON. The intervals against Full history, Prune, and Compact include zero, leaving those smaller differences uncertain.
Table R1 uses GPT-4.1 results on 100 8-objective QA tasks. The paired bootstrap resamples the same task IDs for both methods 10,000 times to estimate the uncertainty in their mean F1 difference. Each method has one recorded execution per task, so the intervals measure uncertainty across tasks, not variation across repeated executions of the same task.
Table R1. ContextRender’s F1 is 53.1. Gains are its F1 minus each comparison method’s F1, in percentage points; each gain has a separate 95% confidence interval.
| Comparison method | Method F1 | Gain (F1 points) | 95% CI for gain |
|---|---|---|---|
| Full history | 50.8 | +2.3 | [−0.03, +4.8] |
| Prune | 51.2 | +1.9 | [−0.7, +4.7] |
| Compact (LLM) | 52.0 | +1.1 | [−1.7, +4.0] |
| Recency + Summary | 48.5 | +4.6 | [+2.3, +7.2] |
| Semantic Retrieval | 46.4 | +6.7 | [+2.8, +10.8] |
| ACON | 48.1 | +5.0 | [+1.0, +9.5] |
Q3a. Sensitivity to Semantic Retrieval’s recent-history cap.
The 60% setting is a ceiling on recent results, not a fixed reservation of context. At the main 6K history budget, it binds in only 5 of 257 managed steps (1.9%) in the audited 30-task AppWorld GPT-4.1 OpenRouter sweep. Moreover, 87% of the tokens in tool results shown in full were selected by semantic similarity. This addresses the concern that the baseline allocates most of its context to recency: in this audited setting, most retained full-result content is selected for relevance.
We checked the QA runs by asking whether the recorded recent results would exceed the same 60% allowance. At 6K, this occurs in only 8 of 5,439 steps for DeepSeek V4 Pro and none of 1,710 for Muse Glimmer 30B. These counts describe how often the cap would constrain the recent results, based on their recorded sizes.
The cap measurements above cover the retriever configurations just described. The local GPT-4.1 QA runs identified in the audit use a different retriever without this 60% cap, so they are not included in these measurements.
Q3b. Cache behavior underlying the cost comparison.
ContextRender’s observed cache-hit rate on AppWorld Test-Normal is 85.3%, compared with 80.3% for Full history, a 5.0-percentage-point advantage. The table reports all methods, showing that ContextRender preserves cache reuse while selecting history.
AppWorld Test-Normal. ρ is the observed fraction of input tokens served from cache. Differences are in percentage points relative to the Full history row.
| Method | Observed ρ | Difference (pp) |
|---|---|---|
| Full history | 80.3% | - |
| Prune | 69.0% | −11.3 |
| Compact | 73.2% | −7.1 |
| Recency + Summary | 72.2% | −8.1 |
| Semantic Retrieval | 85.8% | +5.5 |
| ACON | 74.4% | −5.9 |
| ContextRender | 85.3% | +5.0 |
The paper’s cost estimates apply a shared cache rate across methods. The observed rates above show that ContextRender preserves cache reuse on Test-Normal. They do not by themselves quantify revised cost savings, which also depend on the corresponding token usage.
Q1. Reuse coverage and attribution.
Thank you for the thoughtful feedback.
Tool-Flow Analysis (TFA) assigns reuse credit when qualifying values from an earlier tool result reappear in later model outputs or tool calls. In the recorded GPT-4.1 runs, 39.0% of distinct AppWorld tool results and 4.1% of QA results receive a non-zero score, showing that the signal is more common in identifier-heavy AppWorld.
One audited AppWorld task obtains a Spotify playlist URL ending in
/654from a Todoist tool result, then usesplaylist_id=654in a later call. The identifier falls below TFA’s six-character threshold and receives no reuse credit. The agent had already copied the URL into an assistant message, so the ID remained available and the task succeeded.Another audited AppWorld task uses
[email protected]. Itsbradley_balltoken appears only in Bradley’s contact result and contributes credit 1. The sharedgmail.comtoken appears in seven results and contributes 1/7 to the earliest, Paul’s contact result. Frequency weighting reduces this ambiguous credit. This specific pattern, where part of a value credits a different result from the full value’s source, accounts for at most 1.3% of AppWorld reuse credits in the audit.The replay audit checks whether values missed by TFA remain available through complete source results or earlier assistant messages. They do in 95.8% of 1,614 AppWorld cases and 99.4% of 681 QA cases. Table 2 separately tests task performance by adding reuse to the same recency-and-relevance selector under the same 6K budget; scores improve on both benchmarks, including QA where fewer results receive reuse credit.