- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces HABIT-Bench, a benchmark for evaluating whether agent memory systems can recover user behavioral defaults from longitudinal evidence and apply them while respecting scope, exceptions, temporal changes, and user endorsement. The benchmark contains 114 synthetic users, 37,152 sessions, and 3,328 four-choice probes across Food, Finance, Software, and Travel. Experiments with eight memory methods and several retrieval and history-compression baselines show limited overall accuracy, with sophisticated memory pipelines providing inconsistent advantages over simpler approaches. The evaluation target is valuable, but the distinction between habit induction and explicit decision reconstruction needs clarification. The diagnostic controls and evaluation interfaces also limit what can be concluded about the causes of failure.
Critical strengths
- The paper addresses an important distinction between retaining user information and determining when that information justifies a personalized decision. Incorporating applicability, exceptions, temporal validity, and endorsement makes the evaluation relevant to practical assistants.
- The distractors represent meaningful memory errors. Outdated workflows, unaccepted suggestions, unresolved decisions, and local exceptions test different ways in which a system can misuse historical information. This is more informative than increasing difficulty solely through unrelated conversational content.
- The controlled habit states and evidence annotations make the intended reasoning traceable. Linking probes to supporting sessions and temporal states provides a useful foundation for inspecting errors and developing more targeted evaluations.
- The human validation goes beyond checking agreement with predetermined labels. Reviewers independently assess answerability and whether the alternatives satisfy the complete request. The identification of four Finance items with missing output requirements demonstrates that the rubric can detect construction defects.
- The baseline coverage is reasonably broad. No-memory, recency, lexical retrieval, dense retrieval, compressed-history, and oracle conditions help contextualize the memory-method results. Including competitive simple baselines is particularly useful for assessing whether additional memory machinery provides a practical benefit.
- The empirical findings challenge the assumption that more elaborate memory systems necessarily improve downstream decisions. The GLM compressed-history control reaches 41.98% macro accuracy, exceeding all evaluated memory methods. The construction-cost analysis adds practical context by showing that inexpensive queries can depend on substantial prior processing.
Critical weaknesses
- The central construct is not sufficiently separated from explicit decision reconstruction. Finance and Software predominantly establish workflows through an earlier ordered shortlist and a later user confirmation referring to an ordinal alternative. These tasks require reference resolution, recovery of an explicitly selected option, and temporal state maintenance. Appendix A.1.5 acknowledges this distinction, but the aggregate results still combine these capabilities under the broader framing of habit induction.
- Some of the longitudinal difficulty comes from deliberately fragmented references whose realism remains insufficiently established. In Figures 7 and 8, users resolve earlier alternatives after hundreds of intervening sessions. Such dependencies are challenging, but their difficulty may depend heavily on retaining an ordered list. Source-derived dialogue content and natural-sounding realizations do not, by themselves, establish that these long-range interaction patterns resemble how users develop and communicate habits.
- The system comparisons depend on both the adaptation interface and the amount of information delivered to the answerer. Seven methods use MedMemoryBench adapters that expose retrieved context to a shared reader, rather than running their original end-to-end interfaces. Effective context lengths also differ substantially; the Qwen3-32B analysis reports approximately 107 tokens for Mem0 and 2,934 for A-MEM. These choices constrain how broadly the results can be interpreted as evidence about the underlying memory systems.
- The evidence metrics do not establish whether a decision is fully supported. Joint A+E requires only one positive evidence hit, which may be insufficient for a probe combining several habits. Clean Grounded excludes retrieved nonbinding evidence, although a record showing that a proposal remained unaccepted can be necessary for rejecting an incorrect action. The paper acknowledges these limitations, but the metrics consequently provide a weaker diagnosis of grounded decision making than the task requires.
- The uniqueness of some action labels needs stronger justification. In Figure 9, the absence of an established trade-off between a coffee maker and a fridge motivates seeking confirmation, but it does not obviously distinguish jointly prioritizing both rooms from provisionally leading with one while retaining the other for confirmation. Since both alternatives defer commitment, the displayed evidence does not fully explain why only one is acceptable. This matters directly for the validity of exact-choice scoring.
Questions for authors
- How do you operationally distinguish habit induction from recovering an explicitly confirmed workflow? Appendix A.1.5 identifies decision reconstruction and policy-state maintenance as the predominant requirements in Finance and Software. What proportion of probes requires inferring a behavioral default from repeated observations, beyond resolving references and updating explicit choices? Relatedly, how much difficulty remains if distant ordinal confirmations are replaced with explicit descriptions of the selected workflows while preserving the same temporal changes and exceptions?
- What accuracy does Oracle Habit State achieve when it supplies only the default, applicability conditions, exceptions, and effective dates, without providing the required action? This condition would help determine whether the remaining difficulty lies in recovering the habit state or composing that state with the current request. The present oracle makes those contributions difficult to distinguish.
- Which native mechanisms remain available under the MedMemoryBench adapters, particularly query-time retrieval refinement and coordination across memory stores? Please also clarify the effective context lengths supplied by each method in the primary experiments. This would help establish whether performance differences reflect memory construction, the query interface, or the amount of evidence delivered to the shared answerer.
- How do the evidence annotations distinguish misleading material from necessary negative evidence? A session documenting an unaccepted suggestion may be essential for deciding that the suggestion should not govern a later action, yet its retrieval makes Clean Grounded fail. For multi-habit probes, can the annotations identify whether the retrieved context contains a complete sufficient evidence chain, rather than merely one positive hit?
- For Figure 9, which observation in the full history rules out provisionally prioritizing Aster while retaining Birch for confirmation? Both this alternative and the gold answer defer a final commitment. More generally, when the history does not establish an attribute trade-off, what annotation rule makes one provisional ranking invalid while making joint prioritization uniquely correct?
- Which models and prompts were used to generate the longitudinal histories, perform naturalization, and construct the probes and alternatives? The AI Use Statement discloses generative assistance, but the current PDF does not identify these components sufficiently clearly. Please clarify whether the generation models overlap with the evaluated models, and how the pipeline checks that naturalization preserves endorsement, temporal scope, and exception boundaries. These are central to the validity of the synthetic evidence.
- Full History is listed as an evaluator control in Appendix A.3 and Table 8, and its truncation behavior is described in Appendix A.5.3, but I could not locate its results. Was this control evaluated, and if so, where are its scores reported? Its comparison with Full Memory would help clarify the contribution of history compression.
- Please check the correspondence between dataset versions and bibliographic entries. The Travel source is identified as Taskmaster-2, while Byrne et al. (2019) refers to the Taskmaster-1 paper; the actual Taskmaster-2 release should be identified explicitly. The A-MEM reference is also dated 2026, whereas its official proceedings entry identifies NeurIPS 2025. These appear to be bibliographic issues requiring correction.
- Could the authors verify that the grouped averages in Table 3 are computed before rounding individual method scores? Using the stated probe counts, the Qwen multi-store Software average is 24.41%, rather than 24.42%, and the GLM multi-store Travel average is 51.23%, rather than 51.24%. The published values are consistent with averaging already rounded accuracies. These are minor numerical discrepancies, but the aggregation procedure should be consistent.
- Finally, Appendix A.5.3 describes strict output parsing, retries, and length-limited responses, but does not report the frequency of final generation failures. How many probes remain unresolved after retries, and how do those failures enter the reported accuracy? This is especially relevant because the smaller Qwen experiments use a more permissive parser. Reporting these counts would clarify how much of the observed failure reflects an incorrect decision versus failure to produce an accepted response.
6 comments
Question 2: State-only oracle.
Following your suggestion, we added a User-level State-only control that supplies structured policies without the current probe's required action. It provides user policies, conditions, exceptions, and version histories visible at the cutoff. Probes for the same user at the same visibility cutoff receive identical contexts, without target-habit selection, probe-resolved rules, or choice mappings. General policy actions remain part of the rules. By contrast, the original Oracle Habit State supplies probe-resolved private state together with the required action. We will label it Oracle State + Action to make the distinction explicit.
| Control | Food | Finance | Software | Travel | Macro |
|---|---|---|---|---|---|
| Oracle Evidence | 58.25 | 42.91 | 39.26 | 55.54 | 48.99 |
| User-level State-only | 80.32 | 56.65 | 60.44 | 66.15 | 65.89 |
| Oracle State + Action | 98.41 | 98.54 | 99.26 | 99.23 | 98.86 |
Providing structured policies improves macro accuracy over Oracle Evidence by 16.90 points [15.00, 18.84], yet accuracy remains at 65.89%. The reader must still select and apply the relevant policies. The comparison distinguishes access to supporting evidence, access to structured policies, and access to the resolved action. It measures final choices under these different information conditions, not state-prediction accuracy or the causal contribution of a single field.
Weakness 3 & Question 3: Adapters and effective context.
Thank you for asking us to clarify the implementation scope. We compare memory pipelines under a unified evaluation interface, rather than all native end-to-end agent mechanisms. Section 3.3 defines the shared-answerer protocol, and Appendix A.3 identifies the MedMemoryBench adapters. As noted in §4.2, the results compare complete configured pipelines rather than isolate memory architecture. We will state the enabled operations more explicitly.
Visible sessions enter user-isolated memory. A-MEM searches related notes, Mem0 uses user-scoped search, MemOS searches its text store, MemRL uses value-aware selection without gold rewards, LightMem returns formatted retrieval, and SeCom applies chronological segmentation and compression. Letta uses archival writes and search, with native memorization and query-time tool chaining disabled. MIRIX uses its memory extraction and routing during construction, followed by multi-type embedding retrieval with both per-type and overall retrieval limits. Native query-time send_message coordination is not invoked.
A similar separation of MIRIX retrieval and answering is used in [1], which retrieves explicitly across its memory types before calling an external answer LLM.
For context accounting, §4.1 specifies that construction and budgets vary by method. In the GLM experiments, the Qwen3-8B tokenizer measures budgets only. GLM performs the LLM-dependent memory operations and answering. Appendix A.5.3 details the input-budget safeguard. To make the effective context clear, we now report the memory context actually delivered to the reader, measured with the recorded local budget tokenizer and averaged equally across domain means:
| Method | Qwen tokens | GLM tokens |
|---|---|---|
| Full Memory | 31,072.0 | 30,841.1 |
| A-MEM | 3,385.3 | 3,894.8 |
| Mem0 | 97.0 | 252.6 |
| MemOS | 399.4 | 491.3 |
| MemRL | 1,728.6 | 1,726.3 |
| LightMem | 252.6 | 302.9 |
| SeCom | 2,355.8 | 2,355.8 |
| Letta | 2,610.5 | 2,610.5 |
| MIRIX | 367.7 | 519.4 |
These counts exclude prompt overhead and are not provider billing tokens. No final reader truncation occurred in the primary predictions. The context lengths differ because of each pipeline's selection, compression, and caps, not an additional cut by the reader. A common answering protocol therefore does not imply equal context lengths. GLM–MIRIX includes both preserved and repaired memory-write configurations, as recorded with the outputs. We will include these measurements alongside the adapter descriptions.
Weakness 4 & Question 4: Evidence and chain coverage.
Retrieving an unaccepted suggestion is not necessarily an error. It can be useful negative evidence without establishing a default. The released annotations support this distinction by separating decisive evidence, temporal context, and nonbinding statements. Clean Grounded@5 specifically requires a correct answer, at least one positive evidence hit, and no nonbinding evidence among the top five attributed sessions. Joint A+E requires a correct answer and at least one positive evidence hit. We will state these definitions explicitly and distinguish them from coverage of each habit and of the full annotation set.
Across six Qwen3-32B methods with source attribution, at least one evidence session for every target habit is covered in 0.07–1.54% of Finance probes and 0.15–0.88% of Software probes, considering all valid returned source IDs. Complete annotated decisive-session coverage is zero in these two domains. These statistics measure coverage of the annotations, not the minimal evidence sufficient to answer. Retrieval caps also limit coverage when the annotation set is large. We report decisive, temporal, and nonbinding evidence coverage separately. Per-habit coverage is marked unavailable when a reliable mapping is missing.
Weakness 5 & Question 5: Figure 9 and the supported action.
Thank you for asking us to clarify the decision criterion. The question is whether the history supports putting Aster first, not simply whether a final booking is made. The shared decision prompt in Appendix A.5.3 states: “Do not invent a preference when the evidence is insufficient.” Keeping another alternative available does not, by itself, justify assigning Aster higher priority.
The history provides the relevant distinction. Session 002 supports the preference for both a coffee maker and a refrigerator. In Session 074, when the preferred room's workspace was unresolved, the user instructed the assistant not to substitute Shoreline automatically and to return for confirmation. In the fuller history, Session 080 explicitly permits giving up the refrigerator for that earlier Taos stay only, with an instruction not to extend the exception. This is a scoped waiver, not a general preference for coffee equipment over cold storage.
In the current request, neither Aster nor Birch satisfies both equipment preferences, and no applicable waiver authorizes prioritizing Aster. Option A nevertheless adds an Aster-first preference. Keeping Birch available does not supply evidence for that preference. Option D leaves the room trade-off unresolved without asserting equal utilities. It independently selects Terrace, supported by Sessions 011 and 103. Under the stated evidence-supported-action criterion, D is the only alternative combining both decisions without adding an unsupported preference. This is what makes the case challenging: the model must preserve uncertainty about one decision while applying a supported default to the other.
The gold-blind human protocol in §3.4 and Appendix A.2.1 assesses the complete action and allows abstention rather than forcing a label. For this item, multiple reviewers A and B independently selected D and judged both answerability and evidence sufficiency to be satisfied. We will retain Figure 9 and explain more clearly why even a provisional priority must be supported by the history.
Question 6: Generation models, prompts, and validation.
Thank you for requesting these details. We distinguish content generation, preliminary model-assisted checks, and final human validation. We will make these roles clearer alongside the construction procedures in §3.2 and Appendix A.1.
For Travel, gpt-5.6-sol was used for user dossiers, chronological planning, and dialogue realization. The released Finance and Software code uses deterministic templates to construct shortlists, confirmations, exceptions, probes, and alternatives. Qwen3-8B was not used to generate the final benchmark text, including histories, naturalized wording, probes, and alternatives. Its earlier involvement was in candidate exploration and difficulty checks, not final content generation.
Validation checks whether the resulting text preserves the intended evidence. Food naturalization is checked against protected recipe facts and habit evidence. Finance and Software checks cover endorsement, effective dates, local exception scope, policy consistency, and every requested decision component. Travel's authoring constraints prohibit filtering candidates by unstated preferences or turning a trip-specific choice into a recurring policy. Its histories are authored before evidence annotation and recoverability checks. The released templates and authoring constraints provide concrete examples of these safeguards.
Final validation includes independent, gold-blind human review. Reviewers A and B independently assess the request, alternatives, and chronological evidence without access to gold labels, private graphs, system predictions, or each other's judgments. They check answerability, evidence sufficiency, temporal scope, and exception boundaries, and may abstain when the evidence is insufficient. Reviewer C adjudicates disagreements, with judgments frozen before unblinding. Content revisions receive targeted blind review. This process checks not only textual consistency but also whether the public evidence supports the intended answer.
Thank you for your detailed review and for recognizing the importance of deciding when memory justifies personalization. HABIT-Bench tests whether longitudinal evidence justifies personalized action as scope, exceptions, and circumstances change. To address your questions, we completed additional experiments covering 12 domain–condition cells and 10,752 predictions. Below, Qwen denotes Qwen3-235B-A22B and GLM denotes GLM-5.3-Flash. Accuracies are reported as percentages.
Our final quality control uses two independent human reviewers, A and B, with blind adjudication of disagreements by a third reviewer, C. Reviewers see the request, alternatives, and chronological evidence, but not gold labels, private graphs, system predictions, or each other's judgments. They may abstain, and judgments are frozen before unblinding. The rubric covers answerability, evidence, scope, exceptions, option balance, naturalness, provenance, and privacy. Across 1,469 reviewed items, pre-adjudication choice agreement is 99.50% for Food, 100.00% for Finance, 99.68% for Software, and 92.84% for Travel. Each percentage uses the corresponding valid-choice denominator.
We appreciate the opportunity to clarify the task's scope. As stated in §3.1, HABIT-Bench "combines behavioral inference and cross-session decision reconstruction." Both require more than recalling a stated preference. Evidence may leave either the default or its applicability unresolved, and even an endorsed workflow must be linked to its confirmation, effective time, and scope before it can justify a new action.
Finance and Software emphasize reconstruction and policy maintenance. Together, they account for 2,048 of the 3,328 probes (61.54%) and 50% of the macro weight. Food combines establishment, ratified reuse, and contrasts involving boundaries and exceptions. Travel emphasizes rule recovery and composition. These mechanisms broaden the benchmark's coverage rather than partition individual probes into pure induction or confirmation. We will make this distinction more prominent. We use dependency distance to stress-test cross-session reasoning, not to estimate how far apart references typically occur in natural conversations.
To examine the role of reference reconstruction, we added an explicit-confirmation control. We replaced ordinal confirmations with descriptions copied from their public shortlists and aligned the corresponding acknowledgments. Dates, scope, exceptions, probes, options, and labels remain unchanged.
Making confirmations explicit improves aggregate accuracy in both domains and input conditions. Yet even explicit confirmations with annotated evidence yield only 58.77% in Finance and 58.09% in Software. Resolving the reference does not make the complete decision straightforward. All 1,368 Finance and 680 Software probes use the same reader and labels, and Oracle Evidence retains identical session IDs. The control therefore tests sensitivity to confirmation form while keeping the evaluated decisions fixed.
The control uses a fixed budget but is not length-matched. The longer text excludes the oldest complete session from 194 Finance and 196 Software Full History inputs. Effects also vary by probe type, including a decrease in Finance's Oracle Evidence dual-as-of slice (32.94% to 30.98%). The aggregate gains show that explicit confirmation helps overall, while the slice results show that it does not remove every source of difficulty. The benchmark continues to test temporal validity and the combination of multiple policies.
All CIs use 10,000 paired within-domain user-cluster resamples, retaining each user's probes together (30 users in Food, 36 in Finance, 18 in Software, and 30 in Travel). These pointwise intervals capture uncertainty from sampling users with predictions held fixed. Macro weights domains equally.