- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ConteXtNext (CXN), a framework for constructing task-specific context from an evolving external memory for LLM agents. Its three main contributions are:
1.Instead of attaching static scores (e.g., Q-values) directly to memory files, CXN stores historical usage observations in a separate evidence store and tracks explicit transformation lineage (across splits, merges, and revisions) under conservation constraints so historical credit transfers without duplication.
-
It formulates context construction as budgeted set-utility maximization: reconstructing item utility on demand by weighting similar past tasks, combining task rewards with retrospective self-assessment, discounting overlapping memories via directional coverage edges, and dynamically scaling the initial context budget to task demand.
-
Evaluated on AppWorld, CRMArena-Pro, and KernelBench with two LLM backbones, CXN improves task success over no-memory baselines (by up to 18.7, 37.5, and 11.0 percentage points) and outperforms memory baselines in 4 of 6 settings while significantly reducing context token/character consumption.
Critical strengths
- The paper identifies a practical limitation in existing agent memory systems, that attaching static Q-values or importance scores directly to memory units breaks when memories are reorganized (split, merged, or revised). Separating the free-form editable Markdown workspace from an immutable, version-bound evidence store is a well-motivated systems design that supports both flexible knowledge refactoring and historical credit assignment.
- CXN is evaluated under a strict online streaming protocol (without future leakage) across three distinct domains (AppWorld, CRMArena-Pro, and KernelBench) and two LLM backbones. It consistently improves over no-memory baselines (up to +37.5 pp on CRMArena-Pro and +18.7 pp on AppWorld) and outperforms four recent memory baselines (ACE, ReasoningBank, ReMe, MemQ) in 4 of 6 settings, while substantially reducing main-agent initial context size (-68.8% vs. BM25) and context token consumption (-74.8% vs. No Memory on KernelBench).
Critical weaknesses
- The paper does not isolate its main methodological contributions: conservation-constrained lineage (), coverage-based redundancy discounting (), and adaptive budgeting (). The only retrieval ablation (
FTS recall) replaces all three simultaneously with BM25 and only drops performance by 4% on AppWorld (76% to 72%). Moreover, the paper never reports how often splits, merges, or revisions actually occur during the experiments. - Table 2 claims a 75% reduction in token usage, but only counts main-agent context tokens. It omits the LLM token overhead from CXN's background memory operations (retrospective self-assessment, Manager revisions/splits/merges, and LLM reranking), even though Section 4.1 states that total tokens including memory operations were tracked.
- The equations rely on continuous scalars ( for split inheritance and for directional coverage) estimated by prompting an LLM. The paper neither evaluates the sensitivity or calibration of these LLM-generated numbers nor provides the prompts, Manager schemas, or exact lexical supplementation formula needed to reproduce the system.
Questions for authors
- How frequently do knowledge transformations (
split,merge,revise,archive) actually occur across the AppWorld, CRMArena-Pro, and KernelBench streams? Furthermore, can you provide ablations that isolate the individual impact of (a) transformation lineage conservation (), (b) directional coverage discounting (), and (c) adaptive budgeting (), rather than collapsing all three into theFTS recallbaseline? - Section 4.1 notes that "total logged model tokens including memory operations" were tracked, whereas Table 2 only reports main-agent context tokens. What is the total token consumption and wall-clock/API cost of CXN compared to the baselines once background memory operations (retrospective self-assessment, Manager revisions/splits/merges, and LLM reranking) are included?
- How are the continuous split inheritance shares and directional coverage values elicited from the LLM, and how sensitive is CXN to noise or miscalibration in these estimates (e.g., compared to uniform split shares or binary coverage)? Additionally, could you provide the exact formula used to combine lexical relevance with set-utility gain during greedy selection?