- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes ContextWeave, a benchmark to evaluate the effect of memory in LLM agent systems. ContextWeave preserves the temporal structure of real work and designs multiple ways such as controlled longitudinal evaluation for the fair evaluation. It also proposes workspace score and evaluation score to evaluate how completed work is assessed in real-world settings and implicit preferences.
The experiment finds that memory improves both the final workspace and alignment with participant-specific work practices. Some other conclusions obtained from the experiments include:
- The effect of memory may differ for different models
- Model can turn momory into actions may have larger gains.
Critical strengths
- The benchmark focuses on the longitudinal evaluation with real user's work reports and it is a valuable resource for memory evaluation.
- The findings about how memory affects different models are interesting and might be valuable to further work.
Critical weaknesses
- The construction pipeline of the benchmark is not very clear since it is complex. The paper lacks of examples to show the detailed examples of how the pipeline is. The designed evaluation pipeline and evaluation metrics seem comprehensive and reasonable but without any examples, it is hard to understand how the evaluation works.
- Only Codex is used in the experiments and whether the conclusion is robust to other coding agents is not investigated.
Questions for authors
- Could you add some examples about to show how the benchmark is constructed?
3 comments
Thank you for your clarification. One suggestion is that you may add the detailed example into the appendix later. I have updated my score.
We’re glad we could address your concern! We’ll consider including this demo in the camera-ready version. Thank you again for your support!
Thanks for your time!
Thank you for taking the time to review our work. We would like to provide further clarification below.
Data Example
Regarding the benchmark construction pipeline, we apologize that our original description was not sufficiently clear. We provide a concrete example to illustrate the process.
We begin with a collection of worklog revision records, similar to a Git history. For example, suppose the following entries were added at three different points in time:
We use an LLM to cluster these worklog entries into tasks. In this example, the first two entries are identified as belonging to the same task, while the third belongs to a separate task.
Human annotators then use the relevant worklog entries and workspace states to reconstruct a task instruction from the first two entries: “Analyze the quality of the dataset.”
The annotations also specify three types of information:
The original data owners assist with this annotation process, helping us recover information missing from the worklogs.
Because providing a GPU-enabled Docker environment during evaluation would be expensive, we instead use a standby agent with controlled behavior to mock responses from GPU-related APIs.
We then run a construction-stage agent in a controlled Docker environment containing the dataset and the mocked GPU API. The agent can directly perform operations within this environment. After execution, human annotators review the resulting workspace to verify that it represents a valid reconstruction of the original work. If the reconstruction does not pass this review, we introduce additional constraints through further human annotation.
This process produces a task instruction, a Docker environment that supports task execution, and a reference execution trace whose final workspace has been reviewed by human annotators.
During evaluation, consider task 2 as an example. We have the nine reference traces from the human-validated reconstructions of tasks 1, together with the Docker package required to execute task 2. Each memory component first processes the first historical traces. We then execute task 2 using the same harness, with the memory produced by the corresponding component. The configurations therefore differ in the memory made available to the agent. In principle, a component that better preserves and retrieves relevant historical information should help the agent perform better on task 2.
We hope this example makes the construction and evaluation workflow clearer. We will consider adding this example to the main text if you think it's helpful.
Cross agent evaluation
Regarding generalization to other agent harnesses, we have carefully considered your concern and conducted an additional experiment using DeepSeek Harness (DSH) with DeepSeek-v4.1-Flash, moving task execution from the original Codex + GPT configuration to a different harness–model combination.
We randomly sampled 100 tasks from the full set of 568 tasks and evaluated three configurations: None, A-mem as a representative ICL-based memory method, and MemOS as a representative summarization-based memory method. The results are shown below:
A-mem continues to outperform MemOS in this additional configuration, consistent with the ranking observed in the original Codex-based experiments. This provides additional evidence that the observed advantage extends beyond the original configuration, although our conclusion remains limited to the two tested harness–model combinations.
We are also continuing to update the benchmark and plan to report more comprehensive evaluations in future work.
We welcome further discussion and hope these clarifications and additional results address your concerns. If so, we would be grateful if you could reconsider your score.