- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how memory systems can help LLM agents understand changes in a user’s emotions, attitudes, and preferences over time. AffectLedger keeps local observations in an append-only ledger, together with their timestamps and original evidence. When answering a question, it retrieves relevant observations, expands the selection using shared references and temporal neighbors, and reconstructs short claims about their relationships across time. These claims and their supporting evidence are added to the context supplied by an existing memory system. Experiments show improvements across five memory systems on two CloneMem history scales. Further comparisons examine the value of evidence selection, reconstruction, and context compression, with mixed results on additional benchmarks.
Critical strengths
I find the problem important for personalization. A user’s current attitude may be difficult to understand from a few isolated statements, especially when earlier experiences explain a later change. Preserving those observations and bringing them together when needed is a sensible approach.
The paper is well written, and I found the method concrete and easy to follow. The separation between keeping the original evidence and constructing an account for the current question is particularly clear. I also like the organization of the experiments. The research questions make the section easy to follow, and the controls examine several useful aspects of the method. In particular, comparing raw evidence, generic summaries, and relation-aware reconstruction helps distinguish the benefit of finding more evidence from the benefit of changing how it is presented. Testing the supplement with several existing memory systems is also useful.
Critical weaknesses
I am less convinced about the technical novelty. The method combines evidence extraction, retrieval, historical expansion, and prompted reconstruction, but it is not yet clear which design choice provides the main advance over a simpler retrieval-and-summary approach. Table 2 is relevant here: generic reconstruction gives accuracy close to the relation-aware version at both history scales. The relation-aware output is shorter, which is useful, but I would like to understand whether this comes from better preservation of relationships or simply a more concise summarization instruction.
My second concern is how well the method preserves dependencies involving several earlier events. For example, a user’s decision last week might depend on a work experience, a family discussion, and a preference expressed much earlier. These observations may involve different references and may not be close in time. The reconstruction prompt does ask for cross-time relationships, but the paper gives less evidence about whether this kind of branching dependency survives both evidence selection and compression. The result at 500k makes this question more relevant: Selected Raw reaches 90.01% accuracy, compared with 87.51% for Full AffectLedger. The paper discusses omitted or downweighted details in its qualitative analysis. I would like a clearer account of what is lost in these cases and when keeping the raw observations would be preferable.
Questions for authors
What would the authors identify as the main technical contribution beyond retrieval followed by summarization? Given the similar accuracy of Generic Reconstruction and Full, would a generic summary constrained to the same output length retain the same performance?
How does the method handle a later attitude or decision that depends on several earlier events involving different references? Could the authors show an example tracing which observations are selected and which dependencies remain explicit in the final supplement?
At 500k, why does Selected Raw outperform Full? From the existing traces, are the errors mainly caused by omitted details, incorrect relationships, or changes in emphasis during reconstruction?
The validator checks the output format and cited evidence IDs, but these checks do not establish that a reconstructed relationship is correct. Beyond the qualitative examples, have the authors assessed how often the claims accurately preserve the relationships supported by the original observations?
4 comments
3. Dependencies across multiple events and references
The reconstruction operates over a selected evidence sequence rather than requiring all observations to share a single reference. Claims can cite multiple observations from different times, allowing the final account to express relationships among experiences that would otherwise appear separately.
One example is CloneMem-500k 295aa6eb. Historical observations describe the subject's strict professional inspection standards, the reassurance derived from planning and control, and the difficulty created when those habits carry into family relationships.
Two selected observations explicitly state:
“I've been bringing my work habits home with me.”
“This is how I've lived my whole life—having a plan is the only way to feel at peace.”
The reconstructed claims retain both sides of this history: his rule-focused approach has become a source of difficulty at home, while structure and rigor still provide emotional reassurance. A further claim connects his professional confidence to later self-questioning and unease. These relationships are supported by separately cited observations rather than being reduced to a single positive or negative judgment.
The final answer is correct with Full but incorrect with Selected Raw. This example illustrates multi-factor relational integration: the reconstruction expresses interacting implications of a longstanding behavior across professional and family settings, while retaining their distinct evidence sources.
4. Why Selected Raw exceeds Full at 500k, and semantic faithfulness
We inspected all 64 CloneMem-500k questions for which Selected Raw was correct and Full was incorrect, comparing the selected observations, reconstructed claims, Host context, and final answers. Across the 1,001 paired questions, there were 64 Raw-only correct cases versus 39 Full-only correct cases.
The recurring patterns were loss of question-discriminating details, changes in emphasis, and incomplete trajectories. Some disagreements instead reflected insufficient selected evidence, empty reconstruction, or downstream answer selection despite a broadly supported claim. Clear unsupported relational inferences were observed, but they were not the dominant qualitative pattern.
For example, in bf06e1ab, selected evidence explicitly states:
“Success isn't about climbing the highest, but about finding what matters most to you—and then investing in it with a peaceful heart.”
Full compresses this into a more general statement about clarifying life priorities, weakening the distinction needed for the correct answer. Conversely, 4c335e82 illustrates an unsupported comparative inference about whether an earlier or later self is more authentic.
The following cases illustrate distinct outcomes observed in the paired traces:
| Case | Evidence and reconstruction |
|---|---|
| Multi-factor integration | Full preserves the tension between professional rigor as reassurance and its negative spillover into family life. |
| Discriminative-detail loss | Selected evidence explicitly rejects success as merely climbing higher; Full generalizes this into broader life priorities. |
| Unsupported comparison | Full infers that past naivety is closer to the subject's true self, although the cited observations do not establish that comparison. |
These cases reveal two distinct challenges for longitudinal reconstruction: preserving answer-relevant specificity and maintaining faithful relationships between observations. The 64-case inspection characterizes the failure patterns in this Raw-correct/Full-incorrect subset.
We also conducted a claim-level semantic-faithfulness audit on 80 sampled questions (40 per scale). The author manually audited 154 citation-addressable claims from 67 questions against their cited observations. 134/154 (87.0%) were Supported, 17 (11.0%) Unsupported, and 3 (1.9%) Ambiguous. The unsupported claims involved temporal or event-sequence errors (6), unsupported factual or actor attribution (6), causal or motivational links (4), and unsupported changes over time (1). Missing or invalid citations and questions without evaluable claims were recorded separately.
These findings provide a more precise account of reconstruction reliability than answer accuracy alone. AffectLedger can organize distributed histories into compact, evidence-linked longitudinal claims, while the observed failures identify where preserving decisive details and distinguishing supported relationships from plausible interpretations matter most. Retaining revisitable source evidence is therefore an integral part of the memory design, enabling subsequent questions to construct different accounts from the underlying history.
Thanks for the response and the extra analyses. The 64-case trace inspection and the claim-level audit are useful, and I'd like to see both in the paper.
My main concerns are still there, though.
On the contribution, the response explains the design philosophy well, but it doesn't point to a technical piece that clearly beats retrieval plus summarization. The length-capped control doesn't settle this. Cutting a generic summary off at Full's token count will drop whatever comes last, so a large loss is expected. A fairer test would generate the summary under that budget. Since the prompt approach didn't hold length well, a few rounds of regeneration or a shorter generic prompt would do. As it stands, Generic and Full are within about 1 point at both scales.
On multi-event dependencies, the example shows two observations, not the branching case I asked about. The response also notes that the same-reference channel was inactive at 500k and that the temporal expansion effect flips sign across scales. So I still don't see evidence that dependencies spanning different references and distant times survive selection.
On 500k, the trace analysis confirms what I suspected. There are 64 Raw-only correct cases against 39 Full-only, and the main failure is losing question-specific detail during compression. Add the 11% unsupported claims, judged by a single author, and it seems keeping the raw evidence is often the safer choice at larger scales.
I think the direction is reasonable, but I'll keep my score.
Thank you for your detailed follow-up and constructive suggestions. We particularly appreciate your comments on multi-event dependencies, which prompted us to revisit our existing traces and identify a substantially more representative example. We have incorporated this case and the additional analyses into the revised manuscript.
1. Multi-event dependencies
Your suggestion to examine more complex dependencies was particularly helpful. We identified a CloneMem-500k case (fdf1dbb8) in which the selected evidence spans 538 days (August 2022–February 2024) and captures multiple interacting factors across distinct source records:
- August 2022 — Initial hesitation: "I was hesitating back then."
- July 2023 — Family and relationship conflict: "If I choose Chen Hao, I'll feel so guilty toward my parents."
- July 2023 — Competing commitments: "Am I being too greedy, wanting career growth, time for family, and a good relationship with Chen Hao all at once?"
- July 2023 — Priority realignment: "Now that priorities have been realigned, I feel much clearer."
- February 2024 — Later commitment: "Work is important, but being there for you and planning the future with Chen Hao is just as important to me."
Importantly, the reconstruction does not merely summarize these observations independently. One reconstructed claim explicitly states:
Her growing certainty was shaped by confronting competing commitments to Chen Hao, work, and family rather than by an untroubled progression.
This claim links multiple historical observations concerning distinct commitments, while another connects the initial hesitation, subsequent priority realignment, and later commitment. Full answers the question correctly (B), whereas Selected Raw predicts E.
We believe this case provides concrete evidence that the complete selection–reconstruction pathway can retain and organize multi-event relationships spanning different life domains and distant historical moments. It also helps distinguish two questions: whether the complete pathway can preserve such dependencies, as illustrated here, and how much each expansion channel contributes independently, as investigated by our factorial analyses.
2. Methodological contribution and length control
We would also like to clarify the technical distinction underlying AffectLedger. Rather than generating a summary directly from the Host's retrieved context, our method maintains a separate, source-grounded evidence substrate, assembles bounded historical evidence at access time, and constructs evidence-linked relational accounts without persisting those interpretations. The distinction concerns both the memory lifecycle and the construction of downstream longitudinal context.
The same-evidence controls help characterize the contribution of this representation. Full achieves comparable accuracy to Generic Reconstruction with shorter supplements (799 vs. 1,022 tokens at 100k; 926 vs. 1,011 at 500k). The Length-Capped Generic experiment provides an additional evaluation under constrained context budgets. We distinguish this deterministic budget control from generating a new summary under an explicit length constraint; our interpretation of the original same-evidence comparison does not depend on treating the two procedures as equivalent.
3. Reconstruction behavior at 500k
The 500k results reveal an informative asymmetry: 64 Raw-only correct cases and 39 Full-only correct cases. Our inspection of the 64 cases identifies recurring losses of question-discriminating details, shifts in emphasis, and incomplete longitudinal trajectories.
At the same time, Full reduces mean added downstream context by 41.6% relative to Selected Raw. The Full-only cases, together with the multi-event example above, also demonstrate instances in which the organized representation leads to a correct answer that direct evidence exposure does not. These findings characterize the distinct accuracy and context-efficiency properties of raw evidence and reconstructed longitudinal accounts.
Importantly, AffectLedger retains the underlying observations in its persistent Ledger. Reconstruction determines the supplementary context supplied for a particular access, rather than replacing the stored evidence.
4. Semantic faithfulness
The claim-level audit further characterizes reconstruction quality using direct comparisons with cited source evidence. Of 154 evaluable claims, 134 (87.0%) were Supported, 17 (11.0%) Unsupported, and 3 (1.9%) Ambiguous. The analysis also identifies the specific error types, including temporal distortions, factual attribution errors, and unsupported causal relationships.
Your suggestion regarding multi-event dependencies was especially valuable in helping us surface a stronger example from the existing experiments. We hope the revised manuscript provides a clearer and more complete account of the method's empirical contributions.
We appreciate the reviewer's recognition of the problem setting and the distinction between preserving historical evidence and constructing context for the current question!!! We address the four questions below:
1. Technical contribution beyond retrieval and summarization
The central contribution of AffectLedger is a memory architecture that separates what should remain persistent from what should be interpreted for a particular query, together with a systematic investigation of how this separation affects downstream reasoning.
The distinction goes beyond retrieving relevant passages and summarizing them. A conventional retrieval-and-summary pipeline can condense historical information, but the resulting summary does not necessarily distinguish the underlying observations from the interpretation constructed over them. This distinction is particularly important for longitudinal affective memory: the same history may support different accounts depending on whether the question concerns the onset of a change, the persistence of an attitude, or the relationship between multiple experiences. Committing to a single consolidated interpretation risks obscuring observations or relationships needed by subsequent questions.
AffectLedger addresses this through three connected design decisions. First, persistent memory retains source-grounded local observations rather than committing to a continuously updated longitudinal account. Second, historical access assembles a bounded evidence sequence that can recover relevant earlier states and transitions beyond direct semantic matches. Third, reconstruction explicitly organizes the selected observations into evidence-linked claims about persistence, change, contrast, and evolution. These claims are constructed for the current query, remain traceable to their supporting evidence, and supplement the existing Host without replacing its native memory procedures or becoming persistent state themselves.
Importantly, we make these design decisions empirically distinguishable rather than treating the entire process as an opaque augmentation. Our controlled comparisons separately examine ordinary retrieval, assembled raw evidence, generic reconstruction, relation-aware reconstruction, and persistent-account alternatives. They reveal that historical evidence assembly provides substantial utility before reconstruction, while relational reconstruction offers a more compact representation with scale-dependent accuracy trade-offs. Furthermore, the complete pathway improves accuracy across all ten Host–scale comparisons, demonstrating that this form of longitudinal context construction can complement heterogeneous memory architectures.
Thus, the contribution is not an isolated retrieval heuristic or a new summarization prompt. It is a revisitable evidence-based memory design, an explicit query-time interface for longitudinal relationships, and an empirical characterization of how evidence availability, relational representation, and memory lifecycle jointly shape downstream utility.
2. Generic reconstruction and output length
We first attempted to match the output lengths by modifying the Generic Reconstruction prompt to explicitly request the corresponding Full token budget. However, the model did not reliably follow these instructions: in a 30-question pilot at each scale, only 4/30 outputs at 100k and 8/30 at 500k fell within ±20% of their target lengths. We therefore retained the original same-evidence comparison and introduced a deterministic length-capped control to evaluate Generic Reconstruction under the same per-question token budgets as Full.
Specifically, we evaluated Length-Capped Generic Reconstruction across all 186 and 1,001 questions. For each question, the frozen Generic Reconstruction supplement was capped at the corresponding Full supplement's token budget; already shorter supplements were retained unchanged. The Host context, selected evidence, question, and downstream answering procedure remained fixed. Length-Capped Generic achieved 83.87% at 100k and 80.62% at 500k, compared with Full's 91.94% and 87.51%, differences of +8.06 and +6.89 percentage points in Full's favor.
The two comparisons illuminate complementary aspects of context construction. The original Generic Reconstruction comparison measures alternative representations of the same evidence, while the length-capped control examines their utility under tighter per-question context budgets. Together, they show that the organization and preservation of historical information matter alongside the number of tokens delivered to the downstream model.