acceptodds
Under review as a conference paper at ICLR 2027

HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

Abstract

In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.

Primary area:
datasets and benchmarks
Keywords:
Agent Memory, Habit Induction, Memory Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rNTFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HABIT-Bench, a benchmark for evaluating whether agent memory systems can recover user behavioral defaults from longitudinal evidence and apply them while respecting scope, exceptions, temporal changes, and user endorsement. The benchmark contains 114 synthetic users, 37,152 sessions, and 3,328 four-choice probes across Food, Finance, Software, and Travel. Experiments with eight memory methods and several retrieval and history-compression baselines show limited overall accuracy, with sophisticated memory pipelines providing inconsistent advantages over simpler approaches. The evaluation target is valuable, but the distinction between habit induction and explicit decision reconstruction needs clarification. The diagnostic controls and evaluation interfaces also limit what can be concluded about the causes of failure.

Critical strengths
  1. The paper addresses an important distinction between retaining user information and determining when that information justifies a personalized decision. Incorporating applicability, exceptions, temporal validity, and endorsement makes the evaluation relevant to practical assistants.
  2. The distractors represent meaningful memory errors. Outdated workflows, unaccepted suggestions, unresolved decisions, and local exceptions test different ways in which a system can misuse historical information. This is more informative than increasing difficulty solely through unrelated conversational content.
  3. The controlled habit states and evidence annotations make the intended reasoning traceable. Linking probes to supporting sessions and temporal states provides a useful foundation for inspecting errors and developing more targeted evaluations.
  4. The human validation goes beyond checking agreement with predetermined labels. Reviewers independently assess answerability and whether the alternatives satisfy the complete request. The identification of four Finance items with missing output requirements demonstrates that the rubric can detect construction defects.
  5. The baseline coverage is reasonably broad. No-memory, recency, lexical retrieval, dense retrieval, compressed-history, and oracle conditions help contextualize the memory-method results. Including competitive simple baselines is particularly useful for assessing whether additional memory machinery provides a practical benefit.
  6. The empirical findings challenge the assumption that more elaborate memory systems necessarily improve downstream decisions. The GLM compressed-history control reaches 41.98% macro accuracy, exceeding all evaluated memory methods. The construction-cost analysis adds practical context by showing that inexpensive queries can depend on substantial prior processing.
Critical weaknesses
  1. The central construct is not sufficiently separated from explicit decision reconstruction. Finance and Software predominantly establish workflows through an earlier ordered shortlist and a later user confirmation referring to an ordinal alternative. These tasks require reference resolution, recovery of an explicitly selected option, and temporal state maintenance. Appendix A.1.5 acknowledges this distinction, but the aggregate results still combine these capabilities under the broader framing of habit induction.
  2. Some of the longitudinal difficulty comes from deliberately fragmented references whose realism remains insufficiently established. In Figures 7 and 8, users resolve earlier alternatives after hundreds of intervening sessions. Such dependencies are challenging, but their difficulty may depend heavily on retaining an ordered list. Source-derived dialogue content and natural-sounding realizations do not, by themselves, establish that these long-range interaction patterns resemble how users develop and communicate habits.
  3. The system comparisons depend on both the adaptation interface and the amount of information delivered to the answerer. Seven methods use MedMemoryBench adapters that expose retrieved context to a shared reader, rather than running their original end-to-end interfaces. Effective context lengths also differ substantially; the Qwen3-32B analysis reports approximately 107 tokens for Mem0 and 2,934 for A-MEM. These choices constrain how broadly the results can be interpreted as evidence about the underlying memory systems.
  4. The evidence metrics do not establish whether a decision is fully supported. Joint A+E requires only one positive evidence hit, which may be insufficient for a probe combining several habits. Clean Grounded excludes retrieved nonbinding evidence, although a record showing that a proposal remained unaccepted can be necessary for rejecting an incorrect action. The paper acknowledges these limitations, but the metrics consequently provide a weaker diagnosis of grounded decision making than the task requires.
  5. The uniqueness of some action labels needs stronger justification. In Figure 9, the absence of an established trade-off between a coffee maker and a fridge motivates seeking confirmation, but it does not obviously distinguish jointly prioritizing both rooms from provisionally leading with one while retaining the other for confirmation. Since both alternatives defer commitment, the displayed evidence does not fully explain why only one is acceptable. This matters directly for the validity of exact-choice scoring.
Questions for authors
  1. How do you operationally distinguish habit induction from recovering an explicitly confirmed workflow? Appendix A.1.5 identifies decision reconstruction and policy-state maintenance as the predominant requirements in Finance and Software. What proportion of probes requires inferring a behavioral default from repeated observations, beyond resolving references and updating explicit choices? Relatedly, how much difficulty remains if distant ordinal confirmations are replaced with explicit descriptions of the selected workflows while preserving the same temporal changes and exceptions?
  2. What accuracy does Oracle Habit State achieve when it supplies only the default, applicability conditions, exceptions, and effective dates, without providing the required action? This condition would help determine whether the remaining difficulty lies in recovering the habit state or composing that state with the current request. The present oracle makes those contributions difficult to distinguish.
  3. Which native mechanisms remain available under the MedMemoryBench adapters, particularly query-time retrieval refinement and coordination across memory stores? Please also clarify the effective context lengths supplied by each method in the primary experiments. This would help establish whether performance differences reflect memory construction, the query interface, or the amount of evidence delivered to the shared answerer.
  4. How do the evidence annotations distinguish misleading material from necessary negative evidence? A session documenting an unaccepted suggestion may be essential for deciding that the suggestion should not govern a later action, yet its retrieval makes Clean Grounded fail. For multi-habit probes, can the annotations identify whether the retrieved context contains a complete sufficient evidence chain, rather than merely one positive hit?
  5. For Figure 9, which observation in the full history rules out provisionally prioritizing Aster while retaining Birch for confirmation? Both this alternative and the gold answer defer a final commitment. More generally, when the history does not establish an attribute trade-off, what annotation rule makes one provisional ranking invalid while making joint prioritization uniquely correct?
  6. Which models and prompts were used to generate the longitudinal histories, perform naturalization, and construct the probes and alternatives? The AI Use Statement discloses generative assistance, but the current PDF does not identify these components sufficiently clearly. Please clarify whether the generation models overlap with the evaluated models, and how the pipeline checks that naturalization preserves endorsement, temporal scope, and exception boundaries. These are central to the validity of the synthetic evidence.
  7. Full History is listed as an evaluator control in Appendix A.3 and Table 8, and its truncation behavior is described in Appendix A.5.3, but I could not locate its results. Was this control evaluated, and if so, where are its scores reported? Its comparison with Full Memory would help clarify the contribution of history compression.
  8. Please check the correspondence between dataset versions and bibliographic entries. The Travel source is identified as Taskmaster-2, while Byrne et al. (2019) refers to the Taskmaster-1 paper; the actual Taskmaster-2 release should be identified explicitly. The A-MEM reference is also dated 2026, whereas its official proceedings entry identifies NeurIPS 2025. These appear to be bibliographic issues requiring correction.
  9. Could the authors verify that the grouped averages in Table 3 are computed before rounding individual method scores? Using the stated probe counts, the Qwen multi-store Software average is 24.41%, rather than 24.42%, and the GLM multi-store Travel average is 51.23%, rather than 51.24%. The published values are consistent with averaging already rounded accuracies. These are minor numerical discrepancies, but the aggregation procedure should be consistent.
  10. Finally, Appendix A.5.3 describes strict output parsing, retries, and length-limited responses, but does not report the frequency of final generation failures. How many probes remain unresolved after retries, and how do those failures enter the reported accuracy? This is especially relevant because the smaller Qwen experiments use a more permissive parser. Reporting these counts would clarify how much of the observed failure reflects an incorrect decision versus failure to produce an accepted response.
6 comments

Thank you for your detailed review and for recognizing the importance of deciding when memory justifies personalization. HABIT-Bench tests whether longitudinal evidence justifies personalized action as scope, exceptions, and circumstances change. To address your questions, we completed additional experiments covering 12 domain–condition cells and 10,752 predictions. Below, Qwen denotes Qwen3-235B-A22B and GLM denotes GLM-5.3-Flash. Accuracies are reported as percentages.

Our final quality control uses two independent human reviewers, A and B, with blind adjudication of disagreements by a third reviewer, C. Reviewers see the request, alternatives, and chronological evidence, but not gold labels, private graphs, system predictions, or each other's judgments. They may abstain, and judgments are frozen before unblinding. The rubric covers answerability, evidence, scope, exceptions, option balance, naturalness, provenance, and privacy. Across 1,469 reviewed items, pre-adjudication choice agreement is 99.50% for Food, 100.00% for Finance, 99.68% for Software, and 92.84% for Travel. Each percentage uses the corresponding valid-choice denominator.

Weaknesses 1–2 & Question 1: Construct and explicit confirmations.

We appreciate the opportunity to clarify the task's scope. As stated in §3.1, HABIT-Bench "combines behavioral inference and cross-session decision reconstruction." Both require more than recalling a stated preference. Evidence may leave either the default or its applicability unresolved, and even an endorsed workflow must be linked to its confirmation, effective time, and scope before it can justify a new action.

Finance and Software emphasize reconstruction and policy maintenance. Together, they account for 2,048 of the 3,328 probes (61.54%) and 50% of the macro weight. Food combines establishment, ratified reuse, and contrasts involving boundaries and exceptions. Travel emphasizes rule recovery and composition. These mechanisms broaden the benchmark's coverage rather than partition individual probes into pure induction or confirmation. We will make this distinction more prominent. We use dependency distance to stress-test cross-session reasoning, not to estimate how far apart references typically occur in natural conversations.

To examine the role of reference reconstruction, we added an explicit-confirmation control. We replaced ordinal confirmations with descriptions copied from their public shortlists and aligned the corresponding acknowledgments. Dates, scope, exceptions, probes, options, and labels remain unchanged.

Domain/inputOriginalExplicitChange, pp [95% CI]
Finance/Oracle Evidence42.9158.77+15.86 [12.90, 18.92]
Software/Oracle Evidence39.2658.09+18.82 [15.00, 22.35]
Finance/Full History19.0125.95+6.94 [4.69, 9.38]
Software/Full History19.1240.00+20.88 [16.69, 25.70]

Making confirmations explicit improves aggregate accuracy in both domains and input conditions. Yet even explicit confirmations with annotated evidence yield only 58.77% in Finance and 58.09% in Software. Resolving the reference does not make the complete decision straightforward. All 1,368 Finance and 680 Software probes use the same reader and labels, and Oracle Evidence retains identical session IDs. The control therefore tests sensitivity to confirmation form while keeping the evaluated decisions fixed.

The control uses a fixed budget but is not length-matched. The longer text excludes the oldest complete session from 194 Finance and 196 Software Full History inputs. Effects also vary by probe type, including a decrease in Finance's Oracle Evidence dual-as-of slice (32.94% to 30.98%). The aggregate gains show that explicit confirmation helps overall, while the slice results show that it does not remove every source of difficulty. The benchmark continues to test temporal validity and the combination of multiple policies.

All CIs use 10,000 paired within-domain user-cluster resamples, retaining each user's probes together (30 users in Food, 36 in Finance, 18 in Software, and 30 in Travel). These pointwise intervals capture uncertainty from sampling users with predictions held fixed. Macro weights domains equally.

Question 2: State-only oracle.

Following your suggestion, we added a User-level State-only control that supplies structured policies without the current probe's required action. It provides user policies, conditions, exceptions, and version histories visible at the cutoff. Probes for the same user at the same visibility cutoff receive identical contexts, without target-habit selection, probe-resolved rules, or choice mappings. General policy actions remain part of the rules. By contrast, the original Oracle Habit State supplies probe-resolved private state together with the required action. We will label it Oracle State + Action to make the distinction explicit.

ControlFoodFinanceSoftwareTravelMacro
Oracle Evidence58.2542.9139.2655.5448.99
User-level State-only80.3256.6560.4466.1565.89
Oracle State + Action98.4198.5499.2699.2398.86

Providing structured policies improves macro accuracy over Oracle Evidence by 16.90 points [15.00, 18.84], yet accuracy remains at 65.89%. The reader must still select and apply the relevant policies. The comparison distinguishes access to supporting evidence, access to structured policies, and access to the resolved action. It measures final choices under these different information conditions, not state-prediction accuracy or the causal contribution of a single field.

Weakness 3 & Question 3: Adapters and effective context.

Thank you for asking us to clarify the implementation scope. We compare memory pipelines under a unified evaluation interface, rather than all native end-to-end agent mechanisms. Section 3.3 defines the shared-answerer protocol, and Appendix A.3 identifies the MedMemoryBench adapters. As noted in §4.2, the results compare complete configured pipelines rather than isolate memory architecture. We will state the enabled operations more explicitly.

Visible sessions enter user-isolated memory. A-MEM searches related notes, Mem0 uses user-scoped search, MemOS searches its text store, MemRL uses value-aware selection without gold rewards, LightMem returns formatted retrieval, and SeCom applies chronological segmentation and compression. Letta uses archival writes and search, with native memorization and query-time tool chaining disabled. MIRIX uses its memory extraction and routing during construction, followed by multi-type embedding retrieval with both per-type and overall retrieval limits. Native query-time send_message coordination is not invoked.

A similar separation of MIRIX retrieval and answering is used in [1], which retrieves explicitly across its memory types before calling an external answer LLM.

For context accounting, §4.1 specifies that construction and budgets vary by method. In the GLM experiments, the Qwen3-8B tokenizer measures budgets only. GLM performs the LLM-dependent memory operations and answering. Appendix A.5.3 details the input-budget safeguard. To make the effective context clear, we now report the memory context actually delivered to the reader, measured with the recorded local budget tokenizer and averaged equally across domain means:

MethodQwen tokensGLM tokens
Full Memory31,072.030,841.1
A-MEM3,385.33,894.8
Mem097.0252.6
MemOS399.4491.3
MemRL1,728.61,726.3
LightMem252.6302.9
SeCom2,355.82,355.8
Letta2,610.52,610.5
MIRIX367.7519.4

These counts exclude prompt overhead and are not provider billing tokens. No final reader truncation occurred in the primary predictions. The context lengths differ because of each pipeline's selection, compression, and caps, not an additional cut by the reader. A common answering protocol therefore does not imply equal context lengths. GLM–MIRIX includes both preserved and repaired memory-write configurations, as recorded with the outputs. We will include these measurements alongside the adapter descriptions.

Weakness 4 & Question 4: Evidence and chain coverage.

Retrieving an unaccepted suggestion is not necessarily an error. It can be useful negative evidence without establishing a default. The released annotations support this distinction by separating decisive evidence, temporal context, and nonbinding statements. Clean Grounded@5 specifically requires a correct answer, at least one positive evidence hit, and no nonbinding evidence among the top five attributed sessions. Joint A+E requires a correct answer and at least one positive evidence hit. We will state these definitions explicitly and distinguish them from coverage of each habit and of the full annotation set.

Across six Qwen3-32B methods with source attribution, at least one evidence session for every target habit is covered in 0.07–1.54% of Finance probes and 0.15–0.88% of Software probes, considering all valid returned source IDs. Complete annotated decisive-session coverage is zero in these two domains. These statistics measure coverage of the annotations, not the minimal evidence sufficient to answer. Retrieval caps also limit coverage when the annotation set is large. We report decisive, temporal, and nonbinding evidence coverage separately. Per-habit coverage is marked unavailable when a reliable mapping is missing.

Weakness 5 & Question 5: Figure 9 and the supported action.

Thank you for asking us to clarify the decision criterion. The question is whether the history supports putting Aster first, not simply whether a final booking is made. The shared decision prompt in Appendix A.5.3 states: “Do not invent a preference when the evidence is insufficient.” Keeping another alternative available does not, by itself, justify assigning Aster higher priority.

The history provides the relevant distinction. Session 002 supports the preference for both a coffee maker and a refrigerator. In Session 074, when the preferred room's workspace was unresolved, the user instructed the assistant not to substitute Shoreline automatically and to return for confirmation. In the fuller history, Session 080 explicitly permits giving up the refrigerator for that earlier Taos stay only, with an instruction not to extend the exception. This is a scoped waiver, not a general preference for coffee equipment over cold storage.

In the current request, neither Aster nor Birch satisfies both equipment preferences, and no applicable waiver authorizes prioritizing Aster. Option A nevertheless adds an Aster-first preference. Keeping Birch available does not supply evidence for that preference. Option D leaves the room trade-off unresolved without asserting equal utilities. It independently selects Terrace, supported by Sessions 011 and 103. Under the stated evidence-supported-action criterion, D is the only alternative combining both decisions without adding an unsupported preference. This is what makes the case challenging: the model must preserve uncertainty about one decision while applying a supported default to the other.

The gold-blind human protocol in §3.4 and Appendix A.2.1 assesses the complete action and allows abstention rather than forcing a label. For this item, multiple reviewers A and B independently selected D and judged both answerability and evidence sufficiency to be satisfied. We will retain Figure 9 and explain more clearly why even a provisional priority must be supported by the history.

Question 6: Generation models, prompts, and validation.

Thank you for requesting these details. We distinguish content generation, preliminary model-assisted checks, and final human validation. We will make these roles clearer alongside the construction procedures in §3.2 and Appendix A.1.

For Travel, gpt-5.6-sol was used for user dossiers, chronological planning, and dialogue realization. The released Finance and Software code uses deterministic templates to construct shortlists, confirmations, exceptions, probes, and alternatives. Qwen3-8B was not used to generate the final benchmark text, including histories, naturalized wording, probes, and alternatives. Its earlier involvement was in candidate exploration and difficulty checks, not final content generation.

Validation checks whether the resulting text preserves the intended evidence. Food naturalization is checked against protected recipe facts and habit evidence. Finance and Software checks cover endorsement, effective dates, local exception scope, policy consistency, and every requested decision component. Travel's authoring constraints prohibit filtering candidates by unstated preferences or turning a trip-specific choice into a recurring policy. Its histories are authored before evidence annotation and recoverability checks. The released templates and authoring constraints provide concrete examples of these safeguards.

Final validation includes independent, gold-blind human review. Reviewers A and B independently assess the request, alternatives, and chronological evidence without access to gold labels, private graphs, system predictions, or each other's judgments. They check answerability, evidence sufficiency, temporal scope, and exception boundaries, and may abstain when the evidence is insufficient. Reviewer C adjudicates disagreements, with judgments frozen before unblinding. Content revisions receive targeted blind review. This process checks not only textual consistency but also whether the public evidence supports the intended answer.

Reviewer_PizGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This benchmark (HABIT-Bench) evaluates how well agent memory systems infer habits from weak evidence and apply them to later requests. In particular, they formulated habit memory as the joint problem of inferring behavioral defaults and determining when the evidence supports their use. The benchmark is a four-domain benchmark with chronological histories, annotated habit states and supporting sessions, and human validation of decision recoverability. The evaluation is done by comparing memory systems under a shared answer protocol and providing diagnostic controls for evidence access and downstream decision making.

Critical strengths
  1. The motivation is quite clear and important. Long-horizon application has suffered for subtle deduction without relying on data that is explicitly provided.

  2. Beyond asking whether applying a user's habit, the paper asked a critical question: For an existing habit, is it really worth to be applied? For either yes or no, how shall one justify the decision?

  3. The comparison is extensive that several related benchmarks are discussed. The evaluation is comprehensive

Critical weaknesses

None

Questions for authors

Table 3 reflects that the performance in Finance and Software are significantly behind other domains. May authors give explanations or at least inferences on the reasons?

1 comment

Thank you for your positive assessment. We appreciate your recognition of the importance of justified personalization and the breadth of our evaluation.

Question: Why is performance lower in Finance and Software?

Finance and Software place greater demands on connecting evidence across long histories and resolving several policies together. Three features help explain the observed gap, consistent with the complementary domain design in §3.2 and Appendix A.1.5:

  1. Longer histories and cross-session references. Finance and Software contain 540 sessions per user, compared with 130 in Food and 124–147 in Travel. The model must link a later ordinal confirmation to an earlier shortlist to identify the endorsed workflow. This requires retaining the relevant evidence and connecting it across sessions.
  2. Tracking which policies remain applicable. Replacements, unaccepted suggestions, and local exceptions require distinguishing the active, endorsed policy from plausible but outdated or nonbinding alternatives. Recalling a previously confirmed workflow is therefore insufficient without resolving its scope and temporal validity.
  3. Joint decisions over multiple habits. Every probe in Finance or Software combines two or three habits, whereas Food targets one and Travel has 456 single-habit and 194 two-habit probes. A correct answer must satisfy all relevant habits together. Recovering only one component does not resolve the request.

To examine the role of reference reconstruction, we ran an explicit-confirmation control on all 2,048 probes from Finance and Software. We replaced ordinal references with descriptions of the selected candidates from their public shortlists, preserving probes, alternatives, labels, chronology, and evidence-session IDs. With the same Qwen3-235B-A22B reader, Oracle Evidence accuracy rises from 42.91% to 58.77% in Finance and from 39.26% to 58.09% in Software. The paired gains are +15.86 points (95% user-cluster bootstrap CI [12.90, 18.92]) and +18.82 points [15.00, 22.35]. Expanded confirmations are longer, so this control is not length-matched.

These gains support reference reconstruction as an important contributor to the difficulty. Yet accuracy remains around 58% even with explicit confirmations and annotated supporting sessions. Making the commitment explicit does not settle which combination of policies applies to a new request. This is why HABIT-Bench evaluates whether remembered information justifies the current decision, not just whether relevant history can be retrieved. We will add this explanation and the control results alongside Table 3.

Reviewer_WGneOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HABIT-Bench to evaluate whether LLM-based agent memory systems can infer behavioral defaults from incomplete evidence across long-term interactions and apply them to subsequent decisions according to their scope, exceptions, temporal validity, and user endorsement.

The benchmark constructs synthetic user histories across four domains using task-oriented dialogue seeds and controlled habit states, comprising 114 users, 37152 sessions, and 3328 four-choice tasks. The authors use a shared answerer within each comparison to evaluate eight memory methods, simple retrieval baselines, and a compressed-history control, and provide supplementary evidence through human validation, oracle controls, capability slices, and cost analysis. Experiments show that existing methods achieve limited accuracy on these tasks, richer memory pipelines do not consistently outperform simple retrieval, and supplying annotated evidence directly still leaves substantial decision difficulty.

Critical strengths
  1. The paper does not directly equate “retrieving relevant history” with having sufficient evidence to justify personalization. Incorporating scope, exceptions, temporal validity, and user endorsement into the evaluation can reveal limitations of memory systems that overall accuracy alone may obscure. Therefore, the research question has clear value and practical significance for long-term agent personalization.

  2. The paper provides broad experimental coverage. It includes No Memory and compressed-history controls, simple retrieval baselines, and multiple memory methods, and compares them under two primary model settings.

Critical weaknesses
  1. There remains a lack of clear separation between “habit induction” and the actual task mechanisms. The paper focuses on inferring habits from longitudinal weak evidence, but the key mechanism in Finance and Software is to connect an earlier ordered shortlist with a later user resolution that explicitly refers to an ordinal alternative, and then track subsequent replacements. This type of task largely measures: (1) cross-session reference reconstruction; (2) decision reconstruction based on explicit user confirmation; and (3) policy-state maintenance and temporal-state resolution. These capabilities are important, but they are not entirely equivalent to inferring behavioral defaults from repeated observations without explicit user confirmation. Finance and Software together account for 2,048 of the 3,328 probes, approximately 61.5%, and contribute half of the weight in the four-domain macro accuracy. This mixture therefore directly affects the interpretation of the overall results. The authors explicitly acknowledge in Appendix A.1.5 that these two domains predominantly test cross-session decision reconstruction and policy-state maintenance. This clarification appropriately limits the claims, but does not resolve the distinction between these capabilities in the evaluation.

  2. Oracle Habit State supplies not only the private habit state but also the required action. Therefore, its accuracy of approximately 98%–99% under Qwen3-235B-A22B primarily shows that the answerer can map the supplied action information to the correct alternative; it does not, by itself, establish that the model can reliably apply a correctly recovered habit state to a subsequent request. The comparison with Oracle Evidence changes both the supplied information and the reconstruction requirements, leaving habit-state reconstruction and downstream action selection insufficiently separated. Consequently, the existing experiments remain insufficient to identify the main bottlenecks.

Questions for authors

Please see Weaknesses

4 comments

Thank you for your thoughtful feedback and for recognizing the value of justified personalization and our broad experimental coverage. We address the domain design below and present new oracle results that clarify where the remaining difficulties lie.

Weakness 1: Habit induction and complementary domain coverage.

We appreciate your attention to Appendix A.1.5. The different domain mechanisms are a deliberate choice to broaden coverage. HABIT-Bench tests complementary ways of establishing behavioral defaults and determining when they apply. Our broad habit-induction framing and aggregate results may have made this distinction less apparent. As stated in §3.1, the task “combines behavioral inference and cross-session decision reconstruction.” Section 3.2 likewise describes the domains as complementary evidence structures.

Food combines split decisions, repeated ratified use, and contrasts involving boundaries and exceptions. Travel emphasizes behavioral-rule recovery and composition. Finance and Software focus on reconstructing endorsed workflows and maintaining their temporal validity. Together, they contribute 2,048 of the 3,328 probes (61.54%) and 50% of the domain-macro weight. An explicit endorsement establishes a commitment, but does not settle whether it remains active, applies to a new request, or is overridden by an exception. Including these domains lets HABIT-Bench cover both induction from observations and the maintenance of confirmed policies. We will bring the rationale in Appendix A.1.5 into the main text and distinguish these demands more clearly when discussing the results.

To examine the role of reference reconstruction, we conducted an explicit-confirmation control. We replaced ordinal references in confirmations with descriptions of the selected candidates from their public shortlists. With the same Qwen3-235B-A22B reader, Oracle Evidence accuracy increases from 42.91% to 58.77% in Finance and 39.26% to 58.09% in Software, over all 2,048 probes. The paired gains are +15.86 points (95% CI [12.90, 18.92]) and +18.82 points [15.00, 22.35]. Probes, alternatives, labels, chronology, and evidence-session IDs remain unchanged. The expanded confirmations are longer, so the control is not length-matched.

Importantly, accuracy remains around 58% even with explicit confirmations and annotated supporting sessions. Resolving the ordinal references helps, but does not make the complete decision straightforward. This is the distinction HABIT-Bench is designed to test: recovering an endorsed choice is not enough to determine how it should guide a later request.

Weakness 2: Oracle operations and the remaining decision bottlenecks.

Thank you for this suggestion. We have added a User-level State-only oracle to distinguish access to policy information from access to the current request's resolved action. All three controls use the same Qwen3-235B-A22B reader and the same 3,328 probes:

  1. Oracle Evidence: supply annotated supporting sessions, bypassing retrieval of those sessions. The reader must reconstruct the relevant policies from dialogue evidence and resolve the request.
  2. User-level State-only (new): supply the user's policies, applicability conditions, exceptions, and version histories visible at the cutoff. Probes for the same user at the same visibility cutoff receive the same context, independent of the query, target habit, or alternatives. General policy actions are retained, but no probe-specific rule selection, required action, or choice mapping is supplied. The reader must therefore select and apply the relevant policies.
  3. Oracle State + Action: supply probe-resolved private state together with the required action, as specified in §4.3. The reader maps that privileged information to an alternative. We will use this explicit label for the original Oracle Habit State condition.

Final-choice accuracy (%):

ControlFoodFinanceSoftwareTravelMacro
Oracle Evidence58.2542.9139.2655.5448.99
User-level State-only (new)80.3256.6560.4466.1565.89
Oracle State + Action98.4198.5499.2699.2398.86

The results clarify two remaining challenges:

  • Turning evidence into usable policy information. Providing structured policies raises macro accuracy from 48.99% to 65.89%, a paired gain of 16.90 points [15.00, 18.84] over Oracle Evidence. The reader performs substantially better with structured policies than with the supporting dialogue alone.
  • Selecting and applying policies to the current request. State-only still trails State + Action by 32.97 points [30.89, 35.04]. Even with policy information available, resolving which policies and actions apply remains challenging. Near-ceiling accuracy is reached when the probe-resolved state and required action are supplied together.

These comparisons measure final-choice accuracy under different information conditions. They do not independently score state prediction or decompose errors into additive stages. All intervals use 10,000 paired within-domain user-cluster bootstrap resamples of fixed predictions. Macro accuracy weights domains equally. We will add the new control to Table 4 and explain what each comparison reveals.

Together, the controls make HABIT-Bench more informative for memory-system development. They show why retrieving relevant history or recognizing a commitment is not sufficient for justified personalization, and help distinguish the difficulties of reconstructing policies from those of applying them.

The author's response solved my problem, I will revise the score.

Thank you for carefully reconsidering your evaluation and for taking the time to review our additional experiments and clarifications. We are glad that our response addressed your concerns, and sincerely appreciate your willingness to revise your score. Your constructive feedback has been very helpful in improving our work.

Comments on the paper

3 comments

Dear Reviewers,

The authors have responded to your comments. Could you please review their responses and update your reviews?

AC

AuthorsOpenReview ↗

Summary of Rebuttal (Part 1/2)

Dear Area Chair,

Thank you for your time and consideration. To facilitate your assessment, we summarize the strengths highlighted by the reviewers and our responses to their main concerns. Our rebuttal includes additional experiments comprising 10,752 predictions, together with clarifications of the benchmark design and evaluation protocol.

Summary of Strengths Highlighted by Reviewers

We appreciate the reviewers' recognition of several contributions:

  • A practically important evaluation target. All three reviewers recognize the value of distinguishing remembered information from evidence that justifies a personalized decision.
  • Broad experimental coverage. The reviewers highlight our comparisons across memory methods, simple retrieval baselines, history controls, and oracle conditions.
  • Traceable evidence and meaningful validation. Reviewer rNTF specifically recognizes the informative distractors, annotated evidence, and independent human assessment of answerability.

Key Concerns Raised and Our Responses

1. Additional Experiments and Diagnostic Controls

  • [WGne-W2, rNTF-Q2] Separating policy information from the required action

    We added a User-level State-only oracle that supplies query-independent policies, conditions, exceptions, and version histories visible at the cutoff without the current probe's required action. With Qwen3-235B-A22B on the same 3,328 probes, four-domain macro accuracy is 48.99% with Oracle Evidence, 65.89% with State-only, and 98.86% with State + Action. These final-choice comparisons distinguish access to evidence, structured policies, and resolved actions. They show that selecting and applying policies remains challenging even when structured policy information is available.

  • [WGne-W1, PizG-Question, rNTF-Q1] Difficulty after making confirmations explicit

    We replaced ordinal confirmations with descriptions of the selected workflows while preserving the probes, labels, chronology, and evidence-session IDs. Oracle Evidence accuracy rises from 42.91% to 58.77% in Finance and 39.26% to 58.09% in Software. The expanded text is not length-matched. Nevertheless, substantial decision difficulty remains even with explicit confirmations and supporting evidence, showing that resolving the reference does not resolve the complete task.

  • [rNTF-Q7] Full History comparison

    We completed the Qwen Full History baseline and supplied the previously omitted GLM results. Full Memory improves macro accuracy over budgeted Full History by 2.10 percentage points with Qwen3-235B-A22B and 8.16 percentage points with GLM-5.3-Flash. Gains vary by domain. The comparison supports the value of compression under bounded context while showing that raw history does not remove the benchmark's difficulty.


2. Human Validation and Case Analysis

  • [rNTF-Q5, rNTF-Q6] Evidence-supported answers

    We clarified the existing gold-blind review process, including independent judgments, permitted abstention, and third-reviewer adjudication. Across 1,469 reviewed items, pre-adjudication choice agreement ranges from 92.84% to 100.00% across domains, calculated on valid choices.

    For Figure 9, we explain why even a provisional priority requires supporting evidence. The applicable history does not authorize an Aster-first preference, and both independent reviewers selected D. The case illustrates the need to preserve uncertainty about one decision while applying a supported default to another.


3. Methodology and Design Clarifications

  • [WGne-W1, PizG-Question, rNTF-Q1] Complementary domain mechanisms

    We clarify that HABIT-Bench covers both behavioral inference and cross-session decision reconstruction. Finance and Software add endorsed-workflow reconstruction and temporal policy maintenance to the benchmark's coverage. Their longer histories, cross-session references, and multi-habit decisions help explain the observed performance gap. We will bring the rationale in Appendix A.1.5 into the main text and clarify the role of reference distance as a controlled stress-test variable.

  • [rNTF-Q3] Evaluation interfaces and effective context

    We specify the enabled operations of each configured memory pipeline and report the memory context actually delivered to the shared reader. This makes the comparison interpretable at the pipeline level without treating it as an isolated test of memory architecture or assuming equal context lengths.

  • [rNTF-Q4] Evidence coverage and negative evidence

    We clarify the definitions of Joint A+E and Clean Grounded and report coverage for individual habits and annotated evidence components. We distinguish annotation coverage from a minimal sufficient evidence chain and explain why an unaccepted suggestion can provide useful negative evidence.

AuthorsOpenReview ↗

Summary of Rebuttal (Part 2/2)

  • [rNTF-Q6] Construction and validation roles

    We distinguish final content generation from preliminary model-assisted checks and independent human validation. The response explains how domain-specific templates, authoring constraints, and validation checks address endorsement, temporal scope, and exception boundaries.


4. Reporting and Reproducibility

  • [rNTF-Q8, rNTF-Q9] References and numerical reporting

    We clarify the Taskmaster-2 release citation and the A-MEM publication entry. Recomputing grouped results from integer counts identifies eight display corrections of 0.01 percentage points, with all group-Macro displays unchanged.

  • [rNTF-Q10] Final-answer completeness and parsing

    All expected primary predictions have valid final answers after any recorded retries, with no unresolved probes omitted from scoring. Applying the primary parser's strict rule to 119,808 saved smaller-Qwen responses reproduces every scored choice. These checks distinguish decision accuracy from failures to produce an accepted response.

Together, these additions strengthen HABIT-Bench as a controlled evaluation of whether longitudinal evidence justifies personalized action. We will incorporate the additional results and clarifications into the revised manuscript.


We appreciate your time and effort in evaluating our submission, and we are happy to provide further clarification if anything is unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.