acceptodds
Under review as a conference paper at ICLR 2027

AffectLedger: Preserving Affective Evidence and Reconstructing Longitudinal Context for LLM Agents

Abstract

For memory-augmented LLM agents, retaining a longitudinal history does not guarantee that downstream reasoning receives the relevant evidence or its cross-time relationships. This gap is especially salient in affective histories, where interpreting local expressions often requires relating observations across interactions. We investigate longitudinal context construction through two complementary dimensions: historical evidence availability and cross-time representation. We introduce AffectLedger, a host-modular memory layer that separates persistent evidence preservation from access-time longitudinal context construction. At write time, it retains source-grounded local affective observations in an append-only Ledger. At memory access, it first assembles relevant observations into a bounded historical evidence sequence, then reconstructs compact, evidence-linked claims that make their supported cross-time relationships explicit. These claims supplement the existing memory Host's context without modifying its native memory procedures. Across five heterogeneous Hosts and two CloneMem history scales, AffectLedger improves accuracy in all ten comparisons (+3.10–13.98 percentage points). Controlled studies show that assembled evidence contributes utility before reconstruction, while relation-aware reconstruction produces shorter supplements with accuracy comparable to generic reconstruction. Compared with a persistent-account-plus-raw design, AffectLedger matches accuracy at 100k using 92.3% fewer added downstream tokens, with an accuracy–context trade-off at 500k. Additional benchmarks show task-dependent benefits in integrating and updating distributed histories. These findings highlight the complementary utility of constructing explicit longitudinal context from revisitable evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, agent memory, affective memory, longitudinal reasoning, memory augmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zyGuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how memory systems can help LLM agents understand changes in a user’s emotions, attitudes, and preferences over time. AffectLedger keeps local observations in an append-only ledger, together with their timestamps and original evidence. When answering a question, it retrieves relevant observations, expands the selection using shared references and temporal neighbors, and reconstructs short claims about their relationships across time. These claims and their supporting evidence are added to the context supplied by an existing memory system. Experiments show improvements across five memory systems on two CloneMem history scales. Further comparisons examine the value of evidence selection, reconstruction, and context compression, with mixed results on additional benchmarks.

Critical strengths

I find the problem important for personalization. A user’s current attitude may be difficult to understand from a few isolated statements, especially when earlier experiences explain a later change. Preserving those observations and bringing them together when needed is a sensible approach.

The paper is well written, and I found the method concrete and easy to follow. The separation between keeping the original evidence and constructing an account for the current question is particularly clear. I also like the organization of the experiments. The research questions make the section easy to follow, and the controls examine several useful aspects of the method. In particular, comparing raw evidence, generic summaries, and relation-aware reconstruction helps distinguish the benefit of finding more evidence from the benefit of changing how it is presented. Testing the supplement with several existing memory systems is also useful.

Critical weaknesses

I am less convinced about the technical novelty. The method combines evidence extraction, retrieval, historical expansion, and prompted reconstruction, but it is not yet clear which design choice provides the main advance over a simpler retrieval-and-summary approach. Table 2 is relevant here: generic reconstruction gives accuracy close to the relation-aware version at both history scales. The relation-aware output is shorter, which is useful, but I would like to understand whether this comes from better preservation of relationships or simply a more concise summarization instruction.

My second concern is how well the method preserves dependencies involving several earlier events. For example, a user’s decision last week might depend on a work experience, a family discussion, and a preference expressed much earlier. These observations may involve different references and may not be close in time. The reconstruction prompt does ask for cross-time relationships, but the paper gives less evidence about whether this kind of branching dependency survives both evidence selection and compression. The result at 500k makes this question more relevant: Selected Raw reaches 90.01% accuracy, compared with 87.51% for Full AffectLedger. The paper discusses omitted or downweighted details in its qualitative analysis. I would like a clearer account of what is lost in these cases and when keeping the raw observations would be preferable.

Questions for authors

What would the authors identify as the main technical contribution beyond retrieval followed by summarization? Given the similar accuracy of Generic Reconstruction and Full, would a generic summary constrained to the same output length retain the same performance?

How does the method handle a later attitude or decision that depends on several earlier events involving different references? Could the authors show an example tracing which observations are selected and which dependencies remain explicit in the final supplement?

At 500k, why does Selected Raw outperform Full? From the existing traces, are the errors mainly caused by omitted details, incorrect relationships, or changes in emphasis during reconstruction?

The validator checks the output format and cited evidence IDs, but these checks do not establish that a reconstructed relationship is correct. Beyond the qualitative examples, have the authors assessed how often the claims accurately preserve the relationships supported by the original observations?

4 comments

We appreciate the reviewer's recognition of the problem setting and the distinction between preserving historical evidence and constructing context for the current question!!! We address the four questions below:

1. Technical contribution beyond retrieval and summarization

The central contribution of AffectLedger is a memory architecture that separates what should remain persistent from what should be interpreted for a particular query, together with a systematic investigation of how this separation affects downstream reasoning.
The distinction goes beyond retrieving relevant passages and summarizing them. A conventional retrieval-and-summary pipeline can condense historical information, but the resulting summary does not necessarily distinguish the underlying observations from the interpretation constructed over them. This distinction is particularly important for longitudinal affective memory: the same history may support different accounts depending on whether the question concerns the onset of a change, the persistence of an attitude, or the relationship between multiple experiences. Committing to a single consolidated interpretation risks obscuring observations or relationships needed by subsequent questions.
AffectLedger addresses this through three connected design decisions. First, persistent memory retains source-grounded local observations rather than committing to a continuously updated longitudinal account. Second, historical access assembles a bounded evidence sequence that can recover relevant earlier states and transitions beyond direct semantic matches. Third, reconstruction explicitly organizes the selected observations into evidence-linked claims about persistence, change, contrast, and evolution. These claims are constructed for the current query, remain traceable to their supporting evidence, and supplement the existing Host without replacing its native memory procedures or becoming persistent state themselves.
Importantly, we make these design decisions empirically distinguishable rather than treating the entire process as an opaque augmentation. Our controlled comparisons separately examine ordinary retrieval, assembled raw evidence, generic reconstruction, relation-aware reconstruction, and persistent-account alternatives. They reveal that historical evidence assembly provides substantial utility before reconstruction, while relational reconstruction offers a more compact representation with scale-dependent accuracy trade-offs. Furthermore, the complete pathway improves accuracy across all ten Host–scale comparisons, demonstrating that this form of longitudinal context construction can complement heterogeneous memory architectures.
Thus, the contribution is not an isolated retrieval heuristic or a new summarization prompt. It is a revisitable evidence-based memory design, an explicit query-time interface for longitudinal relationships, and an empirical characterization of how evidence availability, relational representation, and memory lifecycle jointly shape downstream utility.


2. Generic reconstruction and output length

We first attempted to match the output lengths by modifying the Generic Reconstruction prompt to explicitly request the corresponding Full token budget. However, the model did not reliably follow these instructions: in a 30-question pilot at each scale, only 4/30 outputs at 100k and 8/30 at 500k fell within ±20% of their target lengths. We therefore retained the original same-evidence comparison and introduced a deterministic length-capped control to evaluate Generic Reconstruction under the same per-question token budgets as Full.
Specifically, we evaluated Length-Capped Generic Reconstruction across all 186 and 1,001 questions. For each question, the frozen Generic Reconstruction supplement was capped at the corresponding Full supplement's token budget; already shorter supplements were retained unchanged. The Host context, selected evidence, question, and downstream answering procedure remained fixed. Length-Capped Generic achieved 83.87% at 100k and 80.62% at 500k, compared with Full's 91.94% and 87.51%, differences of +8.06 and +6.89 percentage points in Full's favor.
The two comparisons illuminate complementary aspects of context construction. The original Generic Reconstruction comparison measures alternative representations of the same evidence, while the length-capped control examines their utility under tighter per-question context budgets. Together, they show that the organization and preservation of historical information matter alongside the number of tokens delivered to the downstream model.

3. Dependencies across multiple events and references

The reconstruction operates over a selected evidence sequence rather than requiring all observations to share a single reference. Claims can cite multiple observations from different times, allowing the final account to express relationships among experiences that would otherwise appear separately.
One example is CloneMem-500k 295aa6eb. Historical observations describe the subject's strict professional inspection standards, the reassurance derived from planning and control, and the difficulty created when those habits carry into family relationships.
Two selected observations explicitly state:

“I've been bringing my work habits home with me.”
“This is how I've lived my whole life—having a plan is the only way to feel at peace.”

The reconstructed claims retain both sides of this history: his rule-focused approach has become a source of difficulty at home, while structure and rigor still provide emotional reassurance. A further claim connects his professional confidence to later self-questioning and unease. These relationships are supported by separately cited observations rather than being reduced to a single positive or negative judgment.
The final answer is correct with Full but incorrect with Selected Raw. This example illustrates multi-factor relational integration: the reconstruction expresses interacting implications of a longstanding behavior across professional and family settings, while retaining their distinct evidence sources.


4. Why Selected Raw exceeds Full at 500k, and semantic faithfulness

We inspected all 64 CloneMem-500k questions for which Selected Raw was correct and Full was incorrect, comparing the selected observations, reconstructed claims, Host context, and final answers. Across the 1,001 paired questions, there were 64 Raw-only correct cases versus 39 Full-only correct cases.
The recurring patterns were loss of question-discriminating details, changes in emphasis, and incomplete trajectories. Some disagreements instead reflected insufficient selected evidence, empty reconstruction, or downstream answer selection despite a broadly supported claim. Clear unsupported relational inferences were observed, but they were not the dominant qualitative pattern.
For example, in bf06e1ab, selected evidence explicitly states:

“Success isn't about climbing the highest, but about finding what matters most to you—and then investing in it with a peaceful heart.”

Full compresses this into a more general statement about clarifying life priorities, weakening the distinction needed for the correct answer. Conversely, 4c335e82 illustrates an unsupported comparative inference about whether an earlier or later self is more authentic.
The following cases illustrate distinct outcomes observed in the paired traces:

CaseEvidence and reconstruction
Multi-factor integrationFull preserves the tension between professional rigor as reassurance and its negative spillover into family life.
Discriminative-detail lossSelected evidence explicitly rejects success as merely climbing higher; Full generalizes this into broader life priorities.
Unsupported comparisonFull infers that past naivety is closer to the subject's true self, although the cited observations do not establish that comparison.

These cases reveal two distinct challenges for longitudinal reconstruction: preserving answer-relevant specificity and maintaining faithful relationships between observations. The 64-case inspection characterizes the failure patterns in this Raw-correct/Full-incorrect subset.
We also conducted a claim-level semantic-faithfulness audit on 80 sampled questions (40 per scale). The author manually audited 154 citation-addressable claims from 67 questions against their cited observations. 134/154 (87.0%) were Supported, 17 (11.0%) Unsupported, and 3 (1.9%) Ambiguous. The unsupported claims involved temporal or event-sequence errors (6), unsupported factual or actor attribution (6), causal or motivational links (4), and unsupported changes over time (1). Missing or invalid citations and questions without evaluable claims were recorded separately.
These findings provide a more precise account of reconstruction reliability than answer accuracy alone. AffectLedger can organize distributed histories into compact, evidence-linked longitudinal claims, while the observed failures identify where preserving decisive details and distinguishing supported relationships from plausible interpretations matter most. Retaining revisitable source evidence is therefore an integral part of the memory design, enabling subsequent questions to construct different accounts from the underlying history.

Thanks for the response and the extra analyses. The 64-case trace inspection and the claim-level audit are useful, and I'd like to see both in the paper.

My main concerns are still there, though.

On the contribution, the response explains the design philosophy well, but it doesn't point to a technical piece that clearly beats retrieval plus summarization. The length-capped control doesn't settle this. Cutting a generic summary off at Full's token count will drop whatever comes last, so a large loss is expected. A fairer test would generate the summary under that budget. Since the prompt approach didn't hold length well, a few rounds of regeneration or a shorter generic prompt would do. As it stands, Generic and Full are within about 1 point at both scales.

On multi-event dependencies, the example shows two observations, not the branching case I asked about. The response also notes that the same-reference channel was inactive at 500k and that the temporal expansion effect flips sign across scales. So I still don't see evidence that dependencies spanning different references and distant times survive selection.

On 500k, the trace analysis confirms what I suspected. There are 64 Raw-only correct cases against 39 Full-only, and the main failure is losing question-specific detail during compression. Add the 11% unsupported claims, judged by a single author, and it seems keeping the raw evidence is often the safer choice at larger scales.

I think the direction is reasonable, but I'll keep my score.

Thank you for your detailed follow-up and constructive suggestions. We particularly appreciate your comments on multi-event dependencies, which prompted us to revisit our existing traces and identify a substantially more representative example. We have incorporated this case and the additional analyses into the revised manuscript.

1. Multi-event dependencies

Your suggestion to examine more complex dependencies was particularly helpful. We identified a CloneMem-500k case (fdf1dbb8) in which the selected evidence spans 538 days (August 2022–February 2024) and captures multiple interacting factors across distinct source records:

  • August 2022 — Initial hesitation: "I was hesitating back then."
  • July 2023 — Family and relationship conflict: "If I choose Chen Hao, I'll feel so guilty toward my parents."
  • July 2023 — Competing commitments: "Am I being too greedy, wanting career growth, time for family, and a good relationship with Chen Hao all at once?"
  • July 2023 — Priority realignment: "Now that priorities have been realigned, I feel much clearer."
  • February 2024 — Later commitment: "Work is important, but being there for you and planning the future with Chen Hao is just as important to me."

Importantly, the reconstruction does not merely summarize these observations independently. One reconstructed claim explicitly states:

Her growing certainty was shaped by confronting competing commitments to Chen Hao, work, and family rather than by an untroubled progression.

This claim links multiple historical observations concerning distinct commitments, while another connects the initial hesitation, subsequent priority realignment, and later commitment. Full answers the question correctly (B), whereas Selected Raw predicts E.

We believe this case provides concrete evidence that the complete selection–reconstruction pathway can retain and organize multi-event relationships spanning different life domains and distant historical moments. It also helps distinguish two questions: whether the complete pathway can preserve such dependencies, as illustrated here, and how much each expansion channel contributes independently, as investigated by our factorial analyses.

2. Methodological contribution and length control

We would also like to clarify the technical distinction underlying AffectLedger. Rather than generating a summary directly from the Host's retrieved context, our method maintains a separate, source-grounded evidence substrate, assembles bounded historical evidence at access time, and constructs evidence-linked relational accounts without persisting those interpretations. The distinction concerns both the memory lifecycle and the construction of downstream longitudinal context.

The same-evidence controls help characterize the contribution of this representation. Full achieves comparable accuracy to Generic Reconstruction with shorter supplements (799 vs. 1,022 tokens at 100k; 926 vs. 1,011 at 500k). The Length-Capped Generic experiment provides an additional evaluation under constrained context budgets. We distinguish this deterministic budget control from generating a new summary under an explicit length constraint; our interpretation of the original same-evidence comparison does not depend on treating the two procedures as equivalent.

3. Reconstruction behavior at 500k

The 500k results reveal an informative asymmetry: 64 Raw-only correct cases and 39 Full-only correct cases. Our inspection of the 64 cases identifies recurring losses of question-discriminating details, shifts in emphasis, and incomplete longitudinal trajectories.

At the same time, Full reduces mean added downstream context by 41.6% relative to Selected Raw. The Full-only cases, together with the multi-event example above, also demonstrate instances in which the organized representation leads to a correct answer that direct evidence exposure does not. These findings characterize the distinct accuracy and context-efficiency properties of raw evidence and reconstructed longitudinal accounts.

Importantly, AffectLedger retains the underlying observations in its persistent Ledger. Reconstruction determines the supplementary context supplied for a particular access, rather than replacing the stored evidence.

4. Semantic faithfulness

The claim-level audit further characterizes reconstruction quality using direct comparisons with cited source evidence. Of 154 evaluable claims, 134 (87.0%) were Supported, 17 (11.0%) Unsupported, and 3 (1.9%) Ambiguous. The analysis also identifies the specific error types, including temporal distortions, factual attribution errors, and unsupported causal relationships.

Your suggestion regarding multi-event dependencies was especially valuable in helping us surface a stronger example from the existing experiments. We hope the revised manuscript provides a clearer and more complete account of the method's empirical contributions.

Reviewer_AvbCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AffectLedger adds a supplementary memory pathway for reasoning about a user's evolving feelings and preferences. At write time, it stores explicit affective observations with exact source spans, timestamps, and provenance in an append-only Ledger. At access time, semantic retrieval and reference/temporal expansion select up to sixteen observations, which an LLM reconstructs into compact, evidence-linked longitudinal claims alongside the Host's original context.

Across five Hosts and two CloneMem history scales, the method improves accuracy in all ten comparisons. Controlled studies distinguish the value of additional evidence from its representation and show an accuracy-context trade-off. Additional benchmarks show benefits on BEAM, uncertain gains on REALTALK, and mixed or negative transfer elsewhere.

Critical strengths
  1. The modular design is useful and yields consistent gains across different memory Hosts. Preserving local evidence separately from access-time interpretation retains material that future questions may need, while the supplementary interface works without modifying the Host. Accuracy improves for all five Hosts at both history scales (by roughly 3 to 14 percentage points), and nine of the ten paired confidence intervals exclude zero, including all five at 500k. This supports the paper's central claim across different native memory representations, although the comparisons share benchmark populations.

  2. The controls and candid reporting also strengthen trust in the findings. Raw evidence, generic reconstruction, and persistent-consolidation controls help distinguish evidence access from representation. The appendix reports paired uncertainty and correctness transitions, withdraws a positive interaction interpretation after identifying execution failures, and explains the repaired experiment's limitations. It also distinguishes identifier validation from semantic faithfulness. These choices make the narrower findings useful and credible.

Critical weaknesses
  1. The main limitation is that the comparative value of the specialized components is not yet established. Full performs similarly to Generic v2 (Table 13), without a comparison at matched supplement length. At 500k, simply exposing the selected raw evidence (Selected Raw) is more accurate than Full, and the paired confidence interval excludes zero. Full also lacks a statistically established advantage over Extra Retrieval or Budget Raw at this scale. In the repaired factorial, disabling temporal expansion modestly improves the relation-aware condition, although the overlay retains an unresolved fallback (Table 10). The affect-specific write filter is not isolated either. The contribution is therefore better supported as a useful augmentation recipe than as an advantage for each specialized component. The compression comparison also needs qualification. The headline reduction of over 90% at 100k is accurate for the evaluated Fixed+Raw rendering, but that baseline repeats supporting evidence. Relative to Selected Raw, the reduction is roughly half as large, with an accuracy loss at 500k. The counts exclude WRITE and reconstruction computation, and end-to-end costs are unavailable. The authors correctly limit the claim to added answer-time context, but the abstract and introduction should make these boundaries equally clear.

  2. Semantic reliability and transfer are also not yet well characterized. BEAM improves, but REALTALK's semantic gain is uncertain, PersonaMem is numerically worse, ES-MemEval is essentially unchanged, and LoCoMo declines with a confidence interval excluding zero (Table 21). Numerical declines on unanswerable and abstention questions also merit attention, although category-level uncertainty is not reported. The reconstruction prompt excludes uncertainty or insufficiency statements while allowing partial support and empty outputs, and citation-ID checks do not verify semantic support. The descriptive audit of fifteen questions leaves the frequency of unsupported relationships or omitted qualifications unclear. The CloneMem prompt also appends the supplement after the "Correct Option ID" answer marker (Section 6.2.7). It would help to confirm consistent placement across supplement controls and test whether moving it before the marker affects the comparison with Base. This is a protocol clarification, rather than evidence that placement explains the gains.

Questions for authors
  1. Does Full retain an advantage over a length-matched Generic v2 summary or a same-budget semantic-only retrieval control? Please also clarify the evidence for temporal expansion at 500k and for affect-specific filtering over a general-purpose observation ledger.

  2. How are partial and insufficient support conveyed to the answer model? Can a broader audit of existing outputs relate unsupported relationships or omitted qualifications to the unanswerable, abstention, and LoCoMo declines?

3 comments

We appreciate the reviewer's careful distinction between the demonstrated utility of the complete augmentation pathway and the attribution to individual specialized components!!! We address the comparative controls, reliability findings, and implementation protocol directly:

1. Specialized components and length-controlled comparisons

AffectLedger is designed to separate two complementary aspects of longitudinal context construction: making relevant historical evidence available and organizing its supported cross-time relationships for downstream use. Our experiments investigate both the complete pathway and the distinct roles of evidence availability and representation.

Table 2 shows that Selected Raw improves over Base at both 100k and 500k, establishing the utility of historical evidence assembly before reconstruction. On the same evidence, Relation-aware Full achieves accuracy comparable to Generic Reconstruction while producing shorter supplements. At 500k, Selected Raw remains more accurate than Full (90.01% vs. 87.51%), revealing an important trade-off between compact organization and the preservation of question-specific details.

We first attempted to match the output lengths by modifying the Generic Reconstruction prompt to explicitly request the corresponding Full token budget. However, the model did not reliably follow these instructions: in a 30-question pilot at each scale, only 4/30 outputs at 100k and 8/30 at 500k fell within ±20% of their target lengths.

We therefore evaluated Length-Capped Generic Reconstruction across all questions at both scales. Each frozen Generic Reconstruction supplement was capped at the corresponding Full supplement's token budget; shorter supplements were left unchanged. The underlying evidence and downstream answering procedure were fixed. Accuracy was 83.87% at 100k and 80.62% at 500k, versus Full's 91.94% and 87.51%, differences of +8.06 and +6.89 percentage points.

The original same-evidence comparison and the additional length-capped control address complementary questions: the former compares reconstruction policies, while the latter examines information retention under reduced context budgets. Together, they characterize the accuracy–compactness trade-off of supplying longitudinal context.

We also verified that temporal expansion changes the selected evidence, while its factorial effect varies across scales. At 500k, the estimated interaction is −1.60 pp; at 100k, it is +2.69 pp. Thus, the historical expansion policy contributes to the construction of the evidence sequence, while the available accuracy evidence places its independent effect within a scale-dependent range.

The affect-specific WRITE eligibility filter is implemented as described; its incremental effect relative to a resource-matched general-purpose observation ledger is not separately estimated by the current comparisons. The existing experiments similarly distinguish the full selection pathway from the contribution of its individual semantic-retrieval and expansion components.

These findings support the value of studying evidence preservation, selective access, and relational representation as a coordinated context-construction process, with different components affecting information coverage and downstream utility in different ways.

2. Semantic faithfulness and downstream reliability

We conducted a new semantic-faithfulness audit on a frozen sample of 80 CloneMem questions. Of 154 citation-addressable reconstructed claims manually adjudicated by the author, 134 (87.0%) were Supported, 17 (11.0%) Unsupported, and 3 (1.9%) Ambiguous. Unsupported cases included temporal conflation, incorrect factual or actor attribution, unsupported causal explanations, and an unsupported long-term change. Structural citation failures and questions without evaluable claims were tracked separately.

The deterministic validator checks schema and selected evidence-ID membership. The manual audit additionally evaluates whether the reconstructed propositions and relationships are supported by their cited observations, providing a semantic reliability measurement beyond structural validation.

We further inspected all 64 CloneMem-500k cases in which Selected Raw was correct but Full was incorrect. Some failures involved loss of specific motivations, qualifications, or turning points during reconstruction; others involved incomplete selected evidence or incorrect downstream answer selection despite broadly supported reconstructed claims. The analysis distinguishes errors in historical evidence coverage and relational representation from errors in using the resulting context.

Cross-benchmark behavior is heterogeneous. LoCoMo decreases by 0.0478 in its native score, with the decline concentrated in adversarial questions (446 questions; mean change −0.2152). Single-hop and multi-hop categories are approximately unchanged or slightly positive.

The category-level breakdown makes this concentration explicit:

LoCoMo categoryNBase+AffectLedgerΔ
Adversarial4460.78030.5650−0.2152
Open-domain960.25860.2492−0.0094
Temporal3210.38620.3802−0.0060
Single-hop8410.50470.5062+0.0015
Multi-hop2820.29510.3044+0.0093
Overall1,9860.50580.4580−0.0478

The concentration in adversarial questions points to a particular challenge in distinguishing answerable evidence from contextual information that is relevant but insufficient. The category-level data identify where performance declines, while the source-level case analysis provides more specific evidence about individual failure modes.

PersonaMem is numerically lower overall, ES-MemEval F1 is nearly unchanged, and BEAM improves. REALTALK shows a positive semantic-judge difference, with uncertainty spanning zero. Across these evaluations, the strongest positive patterns concern historical integration, updating, and relational interpretation, consistent with the capabilities AffectLedger was designed to support.

3. Partial support, insufficient evidence, and abstention

The reconstruction schema distinguishes supported and partial claims, as well as sufficient, partial, and insufficient overall support. The evaluated renderer presents valid nonempty claims together with supporting evidence, but does not explicitly expose these support labels as downstream confidence or insufficiency markers. An empty supplement follows the Host-only/Base-reuse path.

This separation between internal evidence qualification and downstream context presentation is relevant to abstention. In the complete CloneMem-500k native unanswerable category (N = 96), Base answers 94/96 correctly and Full answers 89/96 correctly.

Inspection of individual disagreements shows cases in which the reconstructed claims provide relevant contextual information but lack evidence for the specific requested fact, yet the downstream model selects a concrete answer. This suggests an answer-stage overcommitment mechanism: the model may treat useful contextual information as sufficient evidence for an answer that the history does not establish.

The result highlights an important distinction between the faithfulness of the reconstructed information and the downstream model's assessment of evidential sufficiency. Retaining support status within the reconstruction process makes these issues separately observable, while explicit propagation of insufficiency information to downstream QA remains a potential design improvement.

4. Supplement placement and efficiency interpretation

We verified that the evaluated CloneMem answer templates append nonempty supplementary context after the literal Correct Option ID: marker, with the shared answer framing retained across the relevant controls. The marker is part of the answer template, not a supplied gold answer.

An additional 80-question placement pilot compared alternative insertion positions. The observed direction differed between 100k and 500k, without a consistent advantage for either position. We report the placement explicitly to make the evaluation protocol reproducible.

Finally, the reported compression figures refer to added answer-time context. Relative to Selected Raw, Full reduces the mean added token count by 48.4% at 100k and 41.6% at 500k. The larger reductions relative to Fixed+Raw, 92.3% and 91.4%, characterize the evaluated memory-lifecycle comparison, including the repeated evidence in the Fixed+Raw rendering.

These figures measure the downstream context budget. WRITE, embedding retrieval, and reconstruction introduce additional upstream computation, so total runtime efficiency is a separate measurement from the answer-time token reductions reported here.

Overall, the results support a host-modular longitudinal context-construction pathway with measurable downstream value. The additional analyses clarify how evidence coverage, compact relational representation, semantic fidelity, and downstream evidence use interact across tasks and history scales, strengthening the paper's account of both its benefits and its observed trade-offs.

Reviewer_oyFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies longitudinal context construction for memory-augmented LLM agents. It addresses the problem that retaining a longitudinal history does not guarantee that downstream reasoning receives relevant historical evidence or its cross-time relationships. This paper proposes AffectLedger, a host-modular memory layer that preserves source-grounded local affective observations in an append-only Ledger and constructs evidence-linked longitudinal claims through selective historical access and relation-aware reconstruction. Experiments across five memory Hosts on CloneMem-100k and CloneMem-500k show accuracy improvements of 3.10–13.98 percentage points.

Critical strengths
  1. The paper addresses an important problem in long-term memory systems.
  2. AffectLedger can be integrated with different memory architectures without modifying their native memory construction or retrieval procedures.
  3. The evaluation covers five memory systems and multiple benchmarks, including controlled comparisons, paired statistical tests, and qualitative analyses.
Critical weaknesses
  1. APEX-MEM [1] already combines append-only storage with query-time temporal resolution, while E-mem [2] reconstructs episodic context from retained historical evidence. FTA-Mem [3] synthesizes Fact-Time-Affect units into structured context, and Hindsight [4] supports temporal retrieval and evolving opinions. These overlaps make AffectLedger’s methodological increment insufficiently clear. An explicit comparison of evidence representations, selection rules, and reconstruction objectives would help establish its contribution. 
  2. Tables 2 and 13 do not establish a consistent accuracy advantage of relation-aware reconstruction over Generic Reconstruction, while Selected Raw performs better at 500k. Because these conditions produce different context lengths, length-matched comparisons are needed to distinguish relational organization from compression policy. Temporal expansion and direct query conditioning also show scale-dependent effects (Tables 10 and 11). A clean factorial using the final reconstruction controls, together with an independent same-reference expansion ablation, would better isolate the components’ contributions. 
  3. The paper lacks a systematic claim-level evaluation of semantic faithfulness. Downstream task results (Tables 4 and 16) measure utility, while the validator checks output structure and evidence identifiers (Section 6.1.3). Neither directly establishes whether reconstructed claims accurately express persistence, change, contrast, and evolution. A structured evaluation beyond the qualitative trace audit in Section 6.4.9 is needed to assess unsupported relationships and omitted decisive evidence. 
  4. The cross-Host gains are not sufficiently disentangled from additional retrieval and reconstruction resources. A-MEM and MemoryBank use MiniLM with top-5 and top-3 retrieval, respectively (Section 6.1.2), whereas AffectLedger adds BGE-M3 retrieval, up to sixteen selected observations, and LLM-based reconstruction (Section 6.1.3). Thus, Table 1 establishes the utility of the complete module but does not isolate the proposed longitudinal organization. Since the Extra Retrieval and Generic Reconstruction controls are restricted to BM25 (Table 2), resource-matched controls on other Hosts would strengthen this attribution. 
  5. The context reductions in Table 3 are sensitive to baseline rendering: Fixed+Raw repeats some supporting evidence already included in the Fixed account (Section 6.4.5). A deduplicated, reasonably compressed baseline would help establish the robustness of the reported percentages. Moreover, the token measurements exclude WRITE and reconstruction computation (Section 6.4.8), so complete cost and latency measurements are needed to assess practical efficiency.
Questions for authors

Please refer to the weaknesses.

6 comments

We thank the reviewer for recognizing the importance of the problem, the host-modular integration, and the breadth of the evaluation!!! The central questions concern what distinguishes AffectLedger from related memory architectures, how its components should be interpreted, and whether the reconstructed relationships are semantically reliable. We address each below:

1. Methodological contribution relative to recent memory systems

Recent memory architectures share several important ingredients, but differ in what remains persistent, when longitudinal interpretation occurs, and how the resulting context is integrated into downstream reasoning.
APEX-MEM maintains source-backed temporal events and facts in an incrementally updated graph, with temporal resolution at query time. E-mem preserves complete episodic contexts and routes queries to agents that reason within activated episodes. FTA-Mem constructs Fact-Time-Affect memory units with source pointers and typed temporal links, then synthesizes retrieved information into structured context. Hindsight combines temporal/entity retrieval with an evolving opinion network and reflective reasoning.
AffectLedger takes a different representation-lifecycle approach: local source-grounded affective observations remain the persistent evidence substrate, while the longitudinal account is constructed only for the current memory-access request. Its bounded evidence assembly and claim-linked reconstruction provide supplementary context alongside an unchanged memory Host, without maintaining the reconstructed account as persistent state.
This distinction addresses a limitation of treating retrieval and summarization as a single operation. A retrieved history can be condensed into a useful summary, but the summary does not necessarily preserve the distinction between historical observations and interpretations derived from them. For longitudinal affective histories, this matters because different questions may require different accounts of the same evidence: one may concern when an attitude changed, another why it persisted, and another how several experiences jointly shaped a later response. Preserving the observations independently of the constructed account allows these relationships to be revisited rather than fixed by an earlier interpretation.
The contribution also extends to making the design decisions empirically distinguishable. Our controlled comparisons examine ordinary retrieval, assembled raw evidence, generic reconstruction, relation-aware reconstruction, and persistent-account alternatives. These comparisons show that historical evidence assembly provides downstream utility before reconstruction, while relation-aware reconstruction offers a compact representation with scale-dependent accuracy trade-offs. Across five heterogeneous Hosts and two history scales, the complete pathway consistently improves accuracy.
Thus, AffectLedger contributes a revisitable evidence-based memory design, a query-time interface for constructing source-linked longitudinal relationships, and a systematic empirical characterization of how evidence availability, representation, and memory lifecycle shape downstream utility. This formulation makes longitudinal context construction an explicit research objective alongside persistent memory organization and retrieval.

2. Length controls and component attribution

Table 2 already separates historical evidence access from its subsequent representation. Selected Raw improves over Base at both scales (88.71% vs. 86.02% at 100k; 90.01% vs. 84.42% at 500k), establishing downstream utility from the assembled evidence before reconstruction. Relation-aware Full achieves 91.94% and 87.51%, compared with 90.86% and 87.81% for Generic Reconstruction over the same selected evidence. The two reconstruction policies therefore achieve comparable accuracy, with Full producing shorter supplements.
We first attempted to match the output lengths by modifying the Generic Reconstruction prompt to explicitly request the corresponding Full token budget. However, the model did not reliably follow these instructions: in a 30-question pilot at each scale, only 4/30 outputs at 100k and 8/30 at 500k fell within ±20% of their target lengths.
We therefore added a complementary Length-Capped Generic Reconstruction control across the complete 186/1,001-question populations. Each frozen Generic Reconstruction supplement was capped at its paired Full token budget, retaining shorter outputs unchanged. All other inputs and downstream answering procedures were held fixed.

ConditionCloneMem-100k (N=186)CloneMem-500k (N=1,001)
Base86.02%84.42%
Selected Raw88.71%90.01%
Generic Reconstruction90.86%87.81%
Length-Capped Generic Reconstruction83.87%80.62%
Relation-aware Full91.94%87.51%

Under the paired context budgets, Full exceeds Length-Capped Generic Reconstruction by +8.06 pp at 100k and +6.89 pp at 500k. These complementary controls distinguish two aspects of the reconstruction design: the original comparison evaluates alternative representations over identical evidence, while the length-capped experiment examines how those representations behave under reduced context budgets. The results highlight compact longitudinal organization as an information-preservation problem, not merely an output-length target.
We also re-audited the temporal-expansion factorial. Its 100k interaction estimate is +2.69 pp; the 500k estimate is −1.60 pp. The results are scale-dependent and do not indicate a consistent independent accuracy gain from this expansion channel. Direct query conditioning also has different numerical effects across the two scales. These analyses help locate the system's measured value in the complete evidence-assembly and reconstruction pathway rather than attributing it uniformly to a single heuristic.
The same-reference channel can contribute in the 100k setting; in the evaluated 500k data, resolved reference identifiers were unavailable, so that expansion channel was inactive. This distinction is reflected in our interpretation of the mechanism results.

3. Claim-level semantic faithfulness

We conducted a new source-level audit on a frozen sample of 80 questions, 40 at each scale. The author manually audited 154 citation-addressable claims from 67 questions, checking their propositions and cross-time relationships directly against cited observations.

ScaleSupportedUnsupportedAmbiguousTotal
CloneMem-100k61 (85.9%)7 (9.9%)3 (4.2%)71
CloneMem-500k73 (88.0%)10 (12.0%)0 (0.0%)83
Total134 (87.0%)17 (11.0%)3 (1.9%)154

Among the unsupported claims, we identified six temporal/event-sequence distortions, six unsupported factual or actor attributions, four unsupported causal or motivational links, and one unsupported change claim. Ten missing/invalid citation entries were separately excluded from semantic adjudication, and 13 sampled questions had no citation-addressable claims.
This analysis directly evaluates the semantic support of reconstructed relationships, complementing the deterministic validator's schema and evidence-ID checks. The 87.0% supported rate measures citation-addressable claims; citation-integrity exclusions and empty outputs are separately accounted for.
A complementary human inspection of all 64 Raw-correct/Full-incorrect questions at 500k further identifies omitted discriminative details, changes in emphasis, and some unsupported relations, while distinguishing reconstruction errors from selection and answering failures. The two analyses assess different aspects of reliability: claim-level support and question-level information preservation.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you very much for your time, careful evaluations, and constructive feedback throughout the discussion. We would like to provide a concise summary of our main contributions and the additional evidence developed during the rebuttal, to facilitate the overall assessment of the paper.

1. Main Contributions

AffectLedger investigates longitudinal context construction as a distinct problem in long-term agent memory. Its central design separates persistent, source-grounded evidence from query-dependent longitudinal interpretation. Rather than maintaining a fixed account of a user's history, AffectLedger preserves revisitable observations, assembles relevant historical evidence at access time, and reconstructs evidence-linked relational claims that supplement an existing memory Host.

The original submission established several findings:

  • Cross-Host complementarity: Positive accuracy gains across five heterogeneous memory Hosts and two CloneMem history scales, covering all ten comparisons (+3.10 to +13.98 percentage points).

  • Evidence availability and representation: Controlled comparisons distinguish the utility of selected historical evidence from its downstream representation. Relation-aware reconstruction achieves accuracy comparable to Generic Reconstruction with shorter supplements.

  • Memory-lifecycle efficiency: Compared with Fixed+Raw, AffectLedger matches accuracy at 100k while using 92.3% fewer added downstream-context tokens, with a scale-dependent accuracy–context trade-off at 500k.

  • Cross-benchmark analysis: Additional evaluations identify positive patterns in historical integration, knowledge updating, and relational interpretation, alongside task-dependent differences.

2. Additional Evidence from the Rebuttal

In response to the reviewers' suggestions, we conducted several further analyses:

(a) Reconstruction and context budgets. We evaluated Length-Capped Generic Reconstruction over the complete paired CloneMem populations, complementing the original same-evidence Generic-versus-Full comparison. We also clarified the measurement scope of downstream-context efficiency and the distinctions between the evaluated memory-lifecycle designs.

(b) Semantic faithfulness. We conducted an human audit of 154 citation-addressable claims from 67 sampled questions. The results comprise 134 Supported (87.0%), 17 Unsupported (11.0%), and 3 Ambiguous (1.9%) claims, with further analysis of temporal, factual-attribution, and relational errors.

(c) Reconstruction behavior at 500k. We inspected all 64 Selected Raw-only correct cases, alongside 39 Full-only correct cases, identifying recurring losses of question-discriminating details, shifts in emphasis, and incomplete longitudinal trajectories. These results provide a more detailed characterization of the information retained and lost during reconstruction.

(d) Multi-event longitudinal integration. Following a particularly helpful reviewer suggestion, we identified an additional CloneMem-500k trace spanning 538 days, involving interconnected relationship, family, and career commitments. The selected observations are organized into evidence-linked claims describing competing priorities and their subsequent realignment. In this case, Full answers correctly whereas Selected Raw does not, providing a concrete illustration of multi-event integration across distant historical records.

3. Revised Manuscript

We have incorporated these findings into the revised manuscript, including the claim-level audit, length-controlled comparison, expanded trace inspection, and multi-event reconstruction case. We also clarified the validator's scope, expanded the comparison with closely related memory systems, and refined the discussion of cross-benchmark behavior.

Together, the original experiments and the additional analyses provide a more comprehensive empirical account of historical evidence availability, longitudinal representation, and their interaction under context constraints.

We sincerely appreciate the reviewers' thoughtful engagement. Their feedback has helped us strengthen both the empirical analysis and the presentation of the work. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.