acceptodds
Under review as a conference paper at ICLR 2027

TaskFid: Benchmarking Agent Memory for Evolving Task-State Fidelity Across Threads and Sessions

Abstract

Long-term memory is essential for agents that collaborate with users across extended, multi-session tasks. Existing benchmarks evaluate retrieval, updating, and temporal reasoning, but rarely directly test evolving task-state fidelity: whether memory supports a correct, consistent, and actionable representation of an ongoing task. In long-horizon collaboration, task state is not a simple accumulation of history: previously valid information may be revised, superseded, or remain applicable only under specific conditions. Relevant history can therefore be retrieved without preserving task-state fidelity, leaving the supported task model outdated, inconsistent, or unable to support subsequent reasoning. We introduce TASKFID to evaluate evolving task-state fidelity across long-horizon task threads and sessions under isolated evolution, independent interleaving, and explicit cross-thread dependencies. At a frozen turn-level checkpoint, which we call an Anchor, independent queries evaluate two capability families: Task Reconstruction, including task-state and collaborative artifact reconstruction, and Resolution & Propagation, including dependency-conditioned actionability and scoped change-impact analysis. Longitudinal families revisit the same target across Anchors. Together, these evaluations measure whether memory supports recovery of and reasoning over the current task model throughout its evolution. Experiments across full-history conditions and representative memory systems show that preserving task-state fidelity remains challenging across task structures. Better retrieval does not consistently translate into higher fidelity, while longitudinal analysis reveals distinct update and retention errors.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, Long-Term Memory, LLM Agents, Task-State Fidelity, Collaborative Task Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_as1BOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TaskFid evaluates whether agent memory preserves and updates task state across sessions and threads, using questions tied to specific points in an evolving task history.

Critical strengths

Frozen-Anchor evaluation tests the same memory state without feeding answers or judge feedback back into memory.

Critical weaknesses
  1. It acknowledges missing independent checks of reference answers and judge accuracy. Appendix D does not measure agreement among independent readers. Wrong references could count correct memory responses as failures. TaskFid needs this check for its own questions. The current scores does not separate memory errors from evaluation errors.

  2. It excludes missing or invalid judge results, which could leave methods scored on different query sets. This concerns judge failures, not automatic removal of badly formatted answers. If difficult cases are excluded unevenly, rankings may change. Equal condition weights do not fix this. The exclusion rule need missing-rate reports, comparisons on shared judged queries, and bounds for unresolved cases.

  3. It scores ten source messages, while the answerer receives all retrieved text. The payment example places useful RAG evidence beyond this cutoff. The retrieval–answer contrast need metrics covering actual answerer input and factual content preserved in summaries.

  4. It credits later-correct unchanged fields even when they were wrong earlier. This mixes preservation with new recovery. TaskFid similarly needs to distinguish retained ability from newly recovered facts. The retention label make this unclear. Report correct-to-correct, correct-to-wrong, and wrong-to-correct field transitions separately to show what was actually retained.

  5. Oracle loses 26.2 in Section 4.4, while Appendix C.8 shows later answers requiring larger maps. Declining scores can therefore reflect harder work rather than forgetting. These curves needs controls for answer size and evidence placement, plus separate tracking of previously correct, unchanged facts.

Questions for authors

Please refer to the weaknesses.

Reviewer_6uerOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TaskFid, a benchmark for evaluating whether agent memory preserves evolving task states across sessions and interleaved task threads. It covers task-state reconstruction, collaborative artifact reconstruction, dependency resolution, and change-impact analysis. Experiments compare full-history, retrieval-based, and persistent-memory systems, with additional analyses of evidence retrieval, task-state fidelity, and longitudinal updates.

Critical strengths

For the strengths,

  • The paper addresses an important problem: whether agent memory can maintain accurate task states as information evolves across sessions.

  • The benchmark covers multiple task structures and includes a reasonable set of memory baselines. The longitudinal analysis provides useful insights into update and retention failures.

Critical weaknesses

For the weaknesses,

  • Recent benchmarks such as StateMemBench, MemOps, RealMem, and MemoryArena already investigate evolving states, memory updates, or interdependent tasks. TaskFid combines these aspects into a unified task-state evaluation framework, but the differences are primarily established through a feature comparison rather than empirical evidence of new evaluation capabilities. It remains unclear which important failure modes TaskFid uniquely captures, or whether it provides substantially different conclusions from existing benchmarks. A more direct comparison with closely related evaluation protocols would strengthen the contribution.

  • Most experiments rely on Qwen3.8-27B as the Answerer, while the full-history setting additionally includes DeepSeek. The paper does not systematically evaluate stronger frontier models such as GPT, Claude, or Gemini, nor does it study performance across model scales. This makes it difficult to distinguish limitations of memory systems from the reasoning limitations of the underlying Answerer, particularly for dependency resolution and change-impact analysis. Moreover, the benchmark relies on synthetic dialogues and author-constructed Gold answers without reporting independent human validation or semantic Judge agreement. Given the complexity of evolving task states, these limitations raise concerns about both the generalizability and reliability of the reported results.

Questions for authors

For the questions,

  • What concrete failure modes does TaskFid reveal that existing benchmarks such as StateMemBench, MemOps, or MemoryArena do not adequately capture? Can the authors provide a more direct empirical comparison rather than only a feature-level comparison?

  • Has the benchmark undergone independent human validation of the Gold answers and semantic Judge? What is the agreement rate across the four query types, particularly for supersession and dependency-related questions?

Please also refer to the weakness part.

Reviewer_yjsGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TaskFid, a benchmark for evaluating whether long-term agent memory supports recovery and use of the current collaborative task model across evolving threads and sessions. It organizes evaluation into task-state and artifact reconstruction, dependency-conditioned actionability, and scoped change-impact analysis, using frozen turn-level Anchors and longitudinal queries. The experiments cover 73 conditions and 5,982 queries across ten authored scenario groups, comparing full-history, retrieval, and persistent-memory systems. The benchmark provides useful structure and diagnostics, but its distinct contribution over recent state-tracking benchmarks and the validity of its synthetic data and LLM-based scoring need stronger support.

Critical strengths
  1. The paper identifies a meaningful distinction between retrieving relevant history and correctly reconstructing the currently valid task state. Its separate analysis of evidence access and answer fidelity helps diagnose where memory systems fail.
  2. TaskFid covers several useful capabilities beyond simple fact recall, including collaborative artifact reconstruction, prerequisites and blockers, and the scope of changes across dependent tasks. Frozen-Anchor evaluation and longitudinal measures make state updates and retention observable over time.
  3. The empirical study is extensive across retrieval and persistent-memory systems. The finding that A-MEM retrieves more reference evidence than RAG while scoring lower on answers is informative, and the Oracle Evidence condition usefully shows that supplying evidence does not eliminate downstream reasoning errors. And the paper is easy to read and understand.
Critical weaknesses
  1. The corpus is built from ten authored scenario groups and dialogues generated with language models, with L2/L3 conditions reusing some source material. Thus, 5,982 queries should not be read as 5,982 independent task scenarios, and performance on this constructed distribution may not generalize to naturally occurring user collaborations.
  2. Semantic answer scores rely on a single Qwen3.8-27B Judge, with no reported independent human calibration or inter-annotator agreement. Since answer correctness is the benchmark’s central outcome, the authors’ own acknowledgment that Gold and Judge reliability still require human calibration leaves system rankings and fine-grained score differences insufficiently validated.
  3. Results are tied to one Answerer setup for all retrieval and memory systems, while full-history runs use either Qwen3.8-27B or DeepSeek Flash and receive up to 262K tokens compared with the standard 40,960-token Answerer context. This makes the full-history gap difficult to attribute specifically to memory quality, and the memory-system rankings may change with the Answerer backbone or context budget.
Questions for authors
  1. What evidence shows that the four TaskFid query types capture the broader construct of evolving task-state fidelity and predict whether an agent can correctly continue the underlying collaboration?
  2. On a stratified sample, how often do human annotators disagree with the Qwen3.8-27B Judge or with the typed Gold labels, and do disagreements affect the relative ranking of methods?
  3. How much of the full-history versus memory-system score gap is explained by the larger full-history context budget, and do memory-system rankings persist with a different Answerer model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.