acceptodds
Under review as a conference paper at ICLR 2027

RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts

Abstract

Large language models and LLM-based agents are widely used as personal chat assistants, enterprise copilots, and autonomous workflow agents. In all these applications, memory (the ability to retain, access, and reason over information accumulated over long contexts and multiple interactions) plays a crucial role in determining the reliability of any agent. We introduce RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark for evaluating compositional reasoning over long contexts. RECON spans 24 case files across three domains (criminal, medical, and financial), each ranging from 50k to 100k tokens, and tests agents on six memory-intensive tasks: reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval. Recent memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed whereas RECON evaluates what happens after the change, whether agents can trace which downstream conclusions are affected, which survive through independent support, and how alternative timelines would have unfolded. Our evaluation reveals substantial limitations across current architectures: no non-Oracle system exceeds Accuracy, with retrieval and reasoning each surfacing as challenges. Our benchmark and code are available at https://anonymous.4open.science/r/RECON-Bench.

Primary area:
datasets and benchmarks
Keywords:
benchmark, agent memory, long context reasoning, LLM agents, compositional reasoning, evaluation, provenance graph, counterfactual reasoning, cascade propagation, retrieval augmented generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_bsqGOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RECON evaluates compositional reasoning over long investigative case files, with 1,604 questions from 24 cases in criminal, medical, and financial domains. Its six task families cover evidence-chain reconstruction, cascading invalidations, source conflicts, counterfactuals, temporal constraints, and temporal fact retrieval. A deterministic generator constructs the underlying evidence dependencies and answer keys before an LLM renders the narrative. The paper compares long-context models, RAG pipelines, and memory systems, together with a structured-graph Oracle condition. On the 1,414-question filtered evaluation set, the best non-Oracle accuracy is 22.4%, compared with 54.6% for the Oracle. Validation includes structural checks, a 200-question human audit, and a 100-response audit of free-form grading.

Critical strengths
  1. The focus on how revisions to evidence affect downstream conclusions is useful. Tracking invalidation and independent support goes beyond retrieving isolated facts, and provides a concrete target for evaluating memory systems.

  2. Constructing evidence graphs and answer keys before generating prose is a sensible design choice. It makes the intended dependencies inspectable and supports targeted diagnostics. The comparison across long-context, retrieval, and memory systems, together with human and grading audits, provides a useful initial characterization.

Critical weaknesses
  1. I am not convinced that the dataset's broader usefulness is sufficiently established. Why were only criminal, medical, and financial investigations selected? These domains share a similar investigation structure. Low model accuracy shows difficulty, it may be better to measure memory capabilities relevant beyond this synthetic setting.

  2. I remain concerned about label reliability. In the 200-question audit, human majority judgments contradicted gold on 8 items and were unsure on 19. How these disagreements and reasonable alternative answers were independently adjudicated is unclear to me. I also find the human–model comparison difficult to interpret: humans received selected evidence packets, whereas long-context models received full case files. A matched-input comparison would be more informative.

  3. Limited variability across cases. The 1,604 questions come from only 24 shared case contexts, so aggregate accuracy may obscure substantial case-level differences. Reporting per-case results would help establish how consistent the findings are.

As an additional presentation clarification, I would like the authors to explain whether Figures 2, 3, and 5 used generative AI and how their text, arrows, and correspondence to the actual pipeline were checked.

Questions for authors

See weaknesses.

Reviewer_JMjMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark for evaluating the compositional reasoning abilities of LLM agents over long contexts. After evidence is invalidated, the agent must trace which downstream conclusions are thereby invalidated and which survive on independent support. RECON contains 24 case files spanning criminal, medical, and financial domains, and includes six types of memory tasks: reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval. The experiments evaluate long-context models, RAG variants, and memory architectures (Mem0, Supermemory, Hindsight, etc.), and introduce a retrieval-perfect Oracle control that receives the ground-truth dependency graph. All non-Oracle systems achieve accuracy no higher than 22.4%, while Oracle reaches only 54.6%. Moreover, Oracle's accuracy on Chain Reconstruction decreases from 100% to 50% as dependency hops increase from 3–4 to 10, indicating that retrieval and compositional depth constitute two mutually independent bottlenecks.

Critical strengths
  • S1. Deterministic generator and coded ground truth. The six-stage seeded pipeline (Blueprint construction → Skeleton expansion → Provenance DAG construction → task synthesis → controlled narration → validation) fully determines the ground truth before narration: all gold answers are derived in code from the skeleton/DAG, and the LLM is responsible only for constrained surface realization and never enters the gold reasoning chain. This avoids the circularity problem common in long-context benchmarks, namely LLM-generated ground truth.
  • S2. Attributable failure diagnosis rather than merely recording failures. The three-part decomposition—Oracle control, per-question gold-evidence recall, and within-task depth sweep—allows errors to be localized to the retrieval side or the reasoning side. The task-level gaps in Table 3 are genuine diagnostic outputs, which is superior to the common leaderboard approach of reporting only aggregate scores.
  • S3. Complete validation pipeline. All 1,604 questions pass checks for DAG acyclicity, temporal monotonicity, and referential integrity (0 cyclic dependencies; 100% temporal monotonicity). Injection-based stress tests detect timestamp flips, DAG corruption, and identifier leakage with 100% detection rate. Closed-book filtering with three models (gpt-5.1, gpt-4.1-mini, gemini-2.5-flash) removes 190 questions that could be guessed from priors, leaving 1,414 questions for the main experiments.
Critical weaknesses
  • W1. Limited novelty at the level of individual tasks. Each of the six task types has precedents. Source Conflict overlaps with TurnaboutLLM (as the paper itself acknowledges, though the latter is limited to a single task); Temporal Fact Retrieval is a classic memory benchmark task already covered by LongMemEval, LoCoMo, MemoryAgentBench, etc. (as the paper's Table 1 itself demonstrates); Temporal Constraint is close to the temporal reasoning component of BEAM; Chain Reconstruction is highly similar to existing long-form multi-hop QA (NovelHopQA, DetectiveQA); and Cascade Propagation and Counterfactual have close counterparts in RippleEdits (ripple effects in knowledge editing) and TRACK (injecting conflicting fact updates into multi-step reasoning chains). The paper's real contribution lies more in the combination of the six task types, the deterministic generation infrastructure, and the failure-attribution framework than in any single new task form. This should be stated more explicitly in the contribution claims.
  • W2. The paper is framed as an "agent memory" benchmark, but the measured focus skews toward compositional reasoning. The Oracle ceiling of only 54.6% means that even if retrieval were solved perfectly, nearly half of the improvement space would remain unrelated to memory/retrieval. The paper's own data support this point: among questions where the gold evidence was fully retrieved, about four-fifths are still answered incorrectly (accuracy 20.6% for RAG-hit questions and 16.7% for memory); according to the Fig. 4 decomposition, roughly half of the failures in both the RAG and memory families occur after full evidence retrieval (RAG 44 vs. 45; memory 43 vs. 49; percentages of questions). Therefore, the low scores of the systems cannot simply be interpreted as "memory system failure"; a substantial portion reflects insufficient downstream reasoning. Framing these numbers as a memory benchmark therefore requires more caution.
  • W3. Oracle is a two-variable manipulation: retrieval and representation format change simultaneously. Oracle removes the narrative retrieval burden, but it also changes the input from narrative text to a structured JSON representation. The representation format is therefore an active variable: on Temporal Fact Retrieval, Oracle (23.5%) is actually 11.8 pp lower than the best non-Oracle system (35.3%), and the authors acknowledge that this is because the structured representation erases the surface narrative details that the task probes. Given this, the "retrieval-perfect ceiling of 54.6%" cannot be interpreted cleanly as a ceiling on reasoning ability, since it is confounded by format compatibility.
  • W4. The human condition and the system test condition are not comparable. Human annotators receive an answer-relevant provenance subgraph filtered per question plus the corresponding narrative excerpt—an input that is cleaner than the full structured graph given to Oracle and very different from the full narrative text faced by the systems. Therefore, 63.3% (on a 60-question sample) is neither human performance under the system test condition (full narrative), nor is the claim that it "exceeds the retrieval-perfect control by 8.7 pp" a comparison within the same input regime. Although the paper acknowledges "a different input format" in the text, it still presents this number as a headline result.
  • W5. There is an unexplained gap between the human baseline and the human validation results. Human Validation (200 questions) reports 86.5% majority agreement with gold; Human Baseline (60 questions) under essentially the same conditions (same answer-relevant evidence packet, three gold-blinded annotators per question, majority vote) achieves only 63.3%, a gap of about 23 pp. Either there are systematic differences between the two studies in annotator pool, time per question, or sampled difficulty; or verifying whether a given answer is correct is substantially easier than solving the question independently. If the latter is the case, the 86.5% agreement rate only shows that gold answers are confirmable, not that they are independently derivable, which would significantly weaken its value as evidence for ground-truth quality.
  • W6. Human validation was never conducted under the full-narrative condition. Both rounds of human study are based on gold evidence packets, and annotators do not see the full case file; therefore, full-text ambiguity or misleading narration introduced by narration cannot be ruled out. The paper also acknowledges that this check covers only a 200-question sample and cannot bound the ambiguity level of the remaining 1,214 questions, and that the deterministic validation targets the skeleton rather than the final released narrative text.
Questions for authors
  • Q1. Oracle and RAG/memory use the same answering backbone (gpt-5.1 and gemini-2.5-flash), yet Oracle reaches only about 55% accuracy; meanwhile, Fig. 4 shows that about half of the memory-family failures occur after full evidence retrieval. Under the narrative that memory/retrieval is the main bottleneck, how do you explain the fact that conditional accuracy remains suppressed by the reasoning layer? Specifically, how do you distinguish between two error modes: "lossy memory storage and noisy recall degrade downstream reasoning" and "the backbone itself has insufficient reasoning capability"?
  • Q2. Human Baseline (63.3%) and Human Validation in Appendix C (86.5%) use essentially the same evidence packets and the same three-annotator, gold-blinded protocol, yet differ by about 23 pp. Please explain: Are there systematic differences in annotator pool, time per question, or sampled difficulty between the two studies? Are the two settings—judging the correctness of a given answer versus independently solving the question—comparable? If the gap cannot be explained, how should the benchmark's claim that its gold standard is reliable be scoped?
3 comments

We have uploaded a revised PDF that incorporates the changes described in our response, marked in blue. We would be grateful to hear whether our response addresses your concerns, and we are happy to clarify anything further.

Thank you for your detailed response. They have resolved my concerns and I will raise my score.

Thank you for engaging so carefully with our responses and for raising your score. Your questions on separating format from retrieval and on the two human studies led to the most informative new analyses in the revision, and the paper is clearer for them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions (revised paper uploaded)

We thank the reviewers and the Area Chair for their constructive feedback. We are encouraged that they found the focus on how revisions to evidence affect downstream conclusions useful, the deterministic generator with code-derived ground truth sound, and the attribution of failures to retrieval or reasoning, across 16 systems, informative.

We have uploaded a revised PDF in which all new or changed text, tables and headings are marked in blue. The main text remains within 10 pages. Because new tables were added to the appendix, some numbers have shifted: the per-domain table (previously Table 11) is now Table 16, and the human baseline appendix (previously D.11) is now D.12.

Structured-graph control (JMjM W3, AC)

  • We renamed the Oracle (also called the retrieval-perfect control) the structured-graph control (Control in the tables and figures), and describe it as a diagnostic condition rather than a retrieval ceiling (§4.1, Table 2 caption, §4.3).
  • We added the analysis that varies input format and retrieval separately, summarized in §4.3 with the full table in Appendix D.6 (Table 11).

Memory versus reasoning (JMjM W2)

  • We added the gold-evidence decomposition on retrieval-hit questions, summarized in §4.3 with the full table in Appendix D.7 (Table 12), and report gold-conditioned accuracy in §4.3 and the Conclusion.

Novelty and positioning (JMjM W1)

  • The contribution statement in §1 now states that the combination of the six tasks with the generator and attribution framework is the core of RECON, and identifies Cascade Propagation and Counterfactual as operations that existing memory benchmarks do not test.
  • We added RippleEdits, BEAM and TurnaboutLLM to Table 1.

Human studies (JMjM W4–W6, bsqG, AC)

  • We replaced the earlier unmatched human–model comparison with a matched-input comparison on the same 60 questions and packets (§4.6, Appendix D.12), and now refer to the result as an evidence-assisted human baseline.
  • We describe the scope and purpose of the evidence packets in each study (§3.3, §4.6, Appendices C and D.12), and state that both studies were gold-blind (Appendix C).
  • We added full-narrative human evaluation and narration outside the gold evidence as limitations (Appendix E).

Label reliability (bsqG, JMjM W5, AC)

  • We added the adjudication protocol and outcomes for all 27 items on which the annotator majority did not match the benchmark answer (Appendix C, Table 5), with a pointer in §3.3.
  • The caption of Table 6 notes that the 4.0% flagged rate also estimates ambiguity among the remaining items.

Case-level variability (bsqG)

  • We added case-clustered bootstrap confidence intervals and leave-one-case-out checks (Appendix D.3, Table 7), and a per-case accuracy table by system family (Table 8).

Domains (bsqG)

  • §3 states that the three domains share one provenance structure by design, §1 notes that the control's lead holds in each domain (Table 16), and Appendix E lists domain scope as a limitation.

Figures (bsqG, AC)

  • Figure 3 is replaced with a high-resolution version.
  • The LLM-use statement now describes the use of an image tool to refine the author-drawn Figures 2, 3 and 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.