acceptodds
Desk rejected at ICLR 2027

ContextWeave: A Real-World Workflow Benchmark

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory evaluation requires benchmarks that capture the temporal structure and cross-task dependencies of real work. We introduce ContextWeave, a longitudinal benchmark comprising 1,005 executable tasks reconstructed from 14 participants’ work records collected over multiple months, including 568 core evaluation tasks. It preserves chronological order, cross-task dependencies, and background tasks while protecting participant privacy. We compare performance on the same target tasks with and without recall under fixed histories and matched initial workspace states. Workspace Score measures task completion and output quality, while Preference Score measures adherence to participant-specific work practices. Across six memory components under a fixed model, the strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. With a fixed memory component, recall improves both scores across all five tested base models. Experience-rich configurations outperform compact summaries and reduce repeated exploration, but also exhibit higher rates of memory-induced execution problems. These findings highlight the need to evaluate memory through downstream outcomes, behavioral effects, and robustness in sustained workflows.

Primary area:
datasets and benchmarks
Keywords:
memory, benchmark, agent, large language model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9z2cOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper proposes ContextWeave, a benchmark to evaluate the effect of memory in LLM agent systems. ContextWeave preserves the temporal structure of real work and designs multiple ways such as controlled longitudinal evaluation for the fair evaluation. It also proposes workspace score and evaluation score to evaluate how completed work is assessed in real-world settings and implicit preferences.

The experiment finds that memory improves both the final workspace and alignment with participant-specific work practices. Some other conclusions obtained from the experiments include:

  1. The effect of memory may differ for different models
  2. Model can turn momory into actions may have larger gains.
Critical strengths
  1. The benchmark focuses on the longitudinal evaluation with real user's work reports and it is a valuable resource for memory evaluation.
  2. The findings about how memory affects different models are interesting and might be valuable to further work.
Critical weaknesses
  1. The construction pipeline of the benchmark is not very clear since it is complex. The paper lacks of examples to show the detailed examples of how the pipeline is. The designed evaluation pipeline and evaluation metrics seem comprehensive and reasonable but without any examples, it is hard to understand how the evaluation works.
  2. Only Codex is used in the experiments and whether the conclusion is robust to other coding agents is not investigated.
Questions for authors
  1. Could you add some examples about to show how the benchmark is constructed?
3 comments
AuthorsOpenReview ↗

Thanks for your time!

Thank you for taking the time to review our work. We would like to provide further clarification below.

Data Example

Regarding the benchmark construction pipeline, we apologize that our original description was not sufficiently clear. We provide a concrete example to illustrate the process.

We begin with a collection of worklog revision records, similar to a Git history. For example, suppose the following entries were added at three different points in time:

[Added at time 1] We analyzed this dataset and found that its quality was poor.

[Added at time 2] The main issue is contamination by advertisements and other noise.

[Added at time 3] Model training encountered an out-of-memory error. The context length is too long, and we are investigating the issue.

We use an LLM to cluster these worklog entries into tasks. In this example, the first two entries are identified as belonging to the same task, while the third belongs to a separate task.

Human annotators then use the relevant worklog entries and workspace states to reconstruct a task instruction from the first two entries: “Analyze the quality of the dataset.”

The annotations also specify three types of information:

  • Task completion criteria: For example, the analysis should identify advertisements as a source of noise in the dataset.
  • Background materials and workspace resources required for execution: In this example, these include the dataset to be analyzed and the required GPU resources.
  • External interfaces required to complete the task.

The original data owners assist with this annotation process, helping us recover information missing from the worklogs.

Because providing a GPU-enabled Docker environment during evaluation would be expensive, we instead use a standby agent with controlled behavior to mock responses from GPU-related APIs.

We then run a construction-stage agent in a controlled Docker environment containing the dataset and the mocked GPU API. The agent can directly perform operations within this environment. After execution, human annotators review the resulting workspace to verify that it represents a valid reconstruction of the original work. If the reconstruction does not pass this review, we introduce additional constraints through further human annotation.

This process produces a task instruction, a Docker environment that supports task execution, and a reference execution trace whose final workspace has been reviewed by human annotators.

During evaluation, consider task 2 as an example. We have the nine reference traces from the human-validated reconstructions of tasks 1, together with the Docker package required to execute task 2. Each memory component first processes the first historical traces. We then execute task 2 using the same harness, with the memory produced by the corresponding component. The configurations therefore differ in the memory made available to the agent. In principle, a component that better preserves and retrieves relevant historical information should help the agent perform better on task 2.

We hope this example makes the construction and evaluation workflow clearer. We will consider adding this example to the main text if you think it's helpful.

Cross agent evaluation

Regarding generalization to other agent harnesses, we have carefully considered your concern and conducted an additional experiment using DeepSeek Harness (DSH) with DeepSeek-v4.1-Flash, moving task execution from the original Codex + GPT configuration to a different harness–model combination.

We randomly sampled 100 tasks from the full set of 568 tasks and evaluated three configurations: None, A-mem as a representative ICL-based memory method, and MemOS as a representative summarization-based memory method. The results are shown below:

MethodScore
None68.23
A-mem81.36
MemOS70.07

A-mem continues to outperform MemOS in this additional configuration, consistent with the ranking observed in the original Codex-based experiments. This provides additional evidence that the observed advantage extends beyond the original configuration, although our conclusion remains limited to the two tested harness–model combinations.

We are also continuing to update the benchmark and plan to report more comprehensive evaluations in future work.

We welcome further discussion and hope these clarifications and additional results address your concerns. If so, we would be grateful if you could reconsider your score.

Thank you for your clarification. One suggestion is that you may add the detailed example into the appendix later. I have updated my score.

We’re glad we could address your concern! We’ll consider including this demo in the camera-ready version. Thank you again for your support!

Reviewer_kgzPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ContextWeave evaluates agent memory through task execution in longitudinal workflows. It reconstructs 1,005 tasks from the work records of 14 participants, including 568 core evaluation tasks. The benchmark compares recall and no-recall conditions using fixed histories and matched starting states. It measures workspace quality, preference adherence, and memory-related behavior.

Critical strengths
  1. The benchmark preserves task order, historical dependencies, and background activity. It evaluates whether memory improves actual work, rather than only whether information can be recalled.
  2. Fixed histories and matched starting states reduce differences caused by earlier execution errors. The evaluation considers both benefits and risks, including reduced exploration and memory-induced problems.
Critical weaknesses
  1. Instructions, resources, and historical trajectories are partly reconstructed. These changes may affect task difficulty and dependencies. Human checks are described, but the paper provides limited quantitative evidence that the final tasks remain faithful to the original work. Workspace and Preference scores also rely on LLM judges. Agreement with independent human judgments would strengthen confidence that these scores measure useful work and genuine preferences.

  2. For GPT-5.5, Table 1 reports no-recall Workspace/Preference scores of 68.08/41.50, while Table 2 reports 67.53/42.07, despite identical mem0 with-recall scores. Some diagnostic results also differ. Separate runs may explain this, but the paper should clarify their relationship. Participant-level results and uncertainty estimates are also needed to show whether the gains are consistent across workflows.

Questions for authors
  1. Can independent human checks validate reconstruction quality and agreement with Workspace/Preference scores?
  2. What explains the differences between Tables 1 and 2? Are the findings stable across participants and repeated runs?
  3. Does experience-rich memory remain better under matched context budgets? Otherwise, please limit this conclusion to the tested configurations, as the paper already acknowledges the confounding factors.
4 comments
AuthorsOpenReview ↗

Thank you for your support! (1/2)

Thank you for your constructive feedback. We have carefully considered your concerns and provide our responses below.

relation between human annotation and LLM evaluate

To address your concern about the relationship between automated scores and human judgments, we conducted an additional human evaluation of the Workspace metric. We recruited two undergraduate annotators with backgrounds in computer science and asked them to independently evaluate the execution results of 120 randomly sampled tasks. We provided the following instructions:

Please evaluate task completion on a 100-point scale. You will be given the task instructions, the initial Docker environment state, the reference Docker environment state after task completion, and the Docker environment state to be evaluated.

Based on the task instructions and the reference state, assess task completion at three levels:

  • Minimum requirements: The conditions necessary for basic task completion.

  • Reference requirements: The level achieved by the reference environment in terms of task fulfillment, correctness, coverage, and supporting verification evidence.

  • Beyond-reference improvements: Reasonable improvements over the reference in completeness, reliability, or thoroughness of verification.

  • Results that fail to meet the minimum requirements should generally receive scores below 60.

  • Results that only meet the minimum requirements should receive approximately 60. > - Results that clearly exceed the minimum requirements but remain below the reference level should generally receive scores between 60 and 79.

  • Results comparable to the reference should receive approximately 80.

  • Results that exceed the reference in important respects may receive scores above 80.

  • Scores above 90 should be reserved for results that substantially surpass the reference in correctness, coverage, verification, or robustness.

Empty files, placeholder implementations, fabricated passing results, hard-coded outputs, deleted tests, and claims of success made only in the worklog without actual deliverables should incur substantial penalties.

The two annotators scored each result independently, and we used the mean of their scores as the human score for that sample. Across the 120 samples, we paired these human scores with the corresponding automated Workspace scores and obtained a Pearson correlation of r = 0.822. This indicates a strong positive correlation between the Workspace metric and human judgments, providing additional evidence supporting the validity of this metric.

About reconstruction quality

As described in Appendix B.3, we invited the original data owner of each reconstructed task to provide a binary judgment of whether the reconstructed content was acceptable. Because these tasks originate from the owners’ own work, they have first-hand knowledge of the original context and requirements and are well positioned to assess whether the reconstruction faithfully reflects their work. We therefore use their judgments as the reference standard for reconstruction quality control.

The mismatch bewtween Table 1 and Table 2

Regarding the numerical inconsistency between Tables 1 and 2, we apologize for the confusion caused by submitting an incorrect PDF version. During our iterative data review, one record was reclassified from mainline to non-mainline, changing the set of records included in the averages and the corresponding denominator. The submitted PDF inadvertently contained results from different aggregation versions.

The discrepancy between Tables 1 and 2 affects only the Workspace and Preference scores for the None baseline. The correct values are those reported in Table 2: 67.53 for Workspace and 42.07 for Preference. Table 1 should therefore report these same values. We apologize for this version inconsistency and the confusion it caused. The human evaluation described above provides additional support for the validity of the Workspace metric, although it does not by itself establish stability across participants or repeated runs.

AuthorsOpenReview ↗

Thank you for your support! (2/2)

About experience-rich memory and memory format

Regarding experience-rich memory and context budgets, thank you for highlighting this potential confounding factor. We further examined the memory-context token counts across the tested configurations:

MethodMemory-context token count
A-mem28,790
LangMem8,275
mem0907
MemoryBank2,150
Memos1,911
Supermemory1,679

A-mem indeed uses substantially more context tokens, which may contribute to its performance. However, MemoryBank and Memos use relatively similar token counts—2,150 and 1,911, respectively—while MemoryBank more often retains examples for in-context learning. In the tested configurations, MemoryBank performs substantially better than the summarization-based methods.

This observation is consistent with the potential benefits of retaining experience-rich examples for downstream tasks. However, this comparison does not isolate memory format from differences in context budget or other methodological factors. We therefore limit our conclusion to the tested configurations: the results suggest that retaining experience-rich examples can be beneficial, but do not establish that this memory format is superior under strictly matched context budgets.

We hope these clarifications address your concerns. Thank you again for your constructive feedback.

Thank you to the authors for the detailed rebuttal. I think it addresses some of my concerns, at least partially. My original rating was 3, and after reading the rebuttal, I feel it could be raised to 3.5, although I'm not quite convinced it reaches a 4. That said, I'm now leaning more toward acceptance. Thank you!

Thank you again for your support of our work! We’re happy to hear that we’ve addressed some of your concerns. Please feel free to let us know if there’s anything we haven’t explained clearly! The only pity is that this year’s ICLR doesn’t have a 3.5 rating option, haha! Thank you once again—we’re very encouraged by your feedback!

Reviewer_To6zOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ContextWeave, a benchmark for evaluating agent memory on long-running real work rather than on isolated tasks or repeated attempts at the same task. The authors take months of document revision histories from 14 participants in an open-source project, segment them into 1,005 tasks with human verification, and reconstruct each one into an executable Docker task with a canonical starting workspace, using 568 of them as evaluation targets. They then run the same target task with and without recalled history, scoring the final workspace and the agent's adherence to each participant's working habits, across six memory components and five base models.

Critical strengths
  • The gap they identify is real and clearly argued: long-context QA tests retrieval, agent benchmarks treat tasks as independent, and self-improvement work repeats the same task, so none of them measure distinct tasks building on one another.
  • Scoring from the final Docker container instead of the agent's own response is the right choice, and the anti-gaming list in the grader prompt (penalize reports claiming success with no artifacts, stubs, hardcoded outputs, weakened tests) is unusually thoughtful.
  • The risk result is the most useful thing in the paper and almost nobody reports it: richer recall helps more and breaks more, with memory-induced problems rising from 0.35% to 7.39% as recall gets richer.
  • The exploration-versus-execution split is a good behavioral diagnostic, because it separates "memory made the trajectory shorter" from "memory moved effort from reconstructing state to doing the work" -- and the per-model results show those come apart.

Overall, I'm positive that this would be a nice addition to practically evaluate agents on real-world tasks.

Critical weaknesses
  • Both headline metrics are LLM-invented and LLM-scored, with no human validation anywhere. Roughly 30 preferences per participant are derived by an LLM, then graded by an LLM; the workspace rubric is written by the grader itself. There is no agreement study, no spot-check accuracy, no human ratings. In Table 1 GPT-5.5 grades GPT-5.5. Tables 1 and 2 also carry no error bars, and Table 1 never states how many trials it averages.
  • Preference Score goes from 41.5 to 70.6, far larger than the Workspace gain, and the rubric set includes writing the record back to the participant's main worklog path. A no-memory agent cannot know that path; a memory-equipped one is simply told it. The judge prompt spends several paragraphs trying to separate placement from style, so the authors clearly saw this risk, yet the split is never reported.
Questions for authors

See the Weaknesses.

3 comments
AuthorsOpenReview ↗

Thanks for your feedback!

Thank you for your constructive feedback. We have carefully considered your concerns and provide our responses below.

LLM scores and human judgments

We value your concern about the relationship between automated scores and human judgments. And we conducted an additional human evaluation of the evaluation metric.

We recruited two undergraduate annotators with backgrounds in computer science and asked them to independently evaluate the execution results of 120 randomly sampled tasks. We provided the following instructions:

Please evaluate task completion on a 100-point scale. You will be given the task instructions, the initial Docker environment state, the reference Docker environment state after task completion, and the Docker environment state to be evaluated.

Based on the task instructions and the reference state, assess task completion at three levels:

  • Minimum requirements: The conditions necessary for basic task completion.

  • Reference requirements: The level achieved by the reference environment in terms of task fulfillment, correctness, coverage, and supporting verification evidence.

  • Beyond-reference improvements: Reasonable improvements over the reference in completeness, reliability, or thoroughness of verification.

  • Results that fail to meet the minimum requirements should generally receive scores below 60.

  • Results that only meet the minimum requirements should receive approximately 60. > - Results that clearly exceed the minimum requirements but remain below the reference level should generally receive scores between 60 and 79.

  • Results comparable to the reference should receive approximately 80.

  • Results that exceed the reference in important respects may receive scores above 80.

  • Scores above 90 should be reserved for results that substantially surpass the reference in correctness, coverage, verification, or robustness.

Empty files, placeholder implementations, fabricated passing results, hard-coded outputs, deleted tests, and claims of success made only in the worklog without actual deliverables should incur substantial penalties.

The two annotators scored each result independently, and we used the mean of their scores as the human score for that sample. Across the 120 samples, we paired these human scores with the corresponding automated Workspace scores and obtained a Pearson correlation of r = 0.822. This indicates a strong positive correlation between the Workspace metric and human judgments, providing additional evidence supporting the validity of this metric.

Preference score

Regarding the composition of the Preference Score, we agree that the contribution of document-writing criteria warrants closer examination. We would first like to clarify that recording work in a user’s customary main worklog is itself part of their preferences. User preferences encompass both writing style and the organization of information across the main worklog and supporting artifacts. These habits affect how well an agent’s outputs fit into the user’s existing workflow.

To further examine this concern, we recomputed the Preference Score after removing document-writing criteria from the rubrics for all tasks whose document-writing criteria scores greater than 0. The resulting scores are:

MethodPreference Score excluding document-writing criteria
None72.95
A-mem78.53
Memos74.63

A-mem retains its advantage after these criteria are removed. This suggests that its improvement in preference adherence is not solely attributable to document-writing criteria and that the comparative conclusion remains consistent under this exclusion.

We hope these clarifications address your concerns. Thank you again for your constructive feedback.

Thanks, my comments are addressed. I'm raising my scores.

Thank you again for your support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.