- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EgoRoutine-Bench, a benchmark for discovering recurring behavioral patterns from long-term egocentric histories without being given the target actions, recurrence conditions, or the number of routines in advance. A routine is represented as an (action, recurrence condition) pair. The benchmark is constructed on top of real EgoLife caption histories using a controlled semi-synthetic procedure that injects synthetic routine events belonging to four predefined condition families: Place, Clock Time, Prior Action, and Frequency. The paper further proposes Routine Memory Induction (RMI), a structured baseline that performs candidate action discovery, occurrence retrieval and verification, context extraction, evidence consolidation, and condition verification to construct compact routine memory. Experiments show that RMI substantially outperforms existing long-term memory agents on joint routine discovery with Qwen3.5-2B/4B, and that the discovered routine memory improves performance on the paper's next-action prediction task.
Critical strengths
1. The problem formulation is meaningful and sufficiently distinct from standard long-term memory evaluation. The paper does not merely ask a model to retrieve the recurrence pattern of a behavior that is already specified in a query. Instead, the system must discover the target actions, their recurrence conditions, and the number of routines jointly. This is a meaningful step beyond query-conditioned behavioral reasoning and is closer to the type of proactive abstraction that a long-term personalized assistant would ideally perform.
2. The controlled benchmark construction and the RMI baseline are both carefully designed. Although the semi-synthetic construction sacrifices some realism, it makes joint action-condition discovery directly measurable, and the authors additionally check background occurrences, alternative conditions, counterfactual evidence dependence, and injection detectability. RMI also provides a reasonably complete structured baseline by decomposing open-ended discovery into candidate action discovery, occurrence verification, context extraction, evidence consolidation, and condition induction. The given-action analysis and memory intervention experiments further suggest that contextual evidence consolidation and correct action-condition pairing contribute meaningfully to performance rather than the gains coming only from maintaining a list of repeated actions.
Critical weaknesses
1. The claimed downstream utility of discovered routine memory is not validated on real future behavior. I agree that a semi-synthetic construction is reasonable for obtaining complete routine ground truth and computing joint precision/recall/F1. However, the paper goes further and argues that discovered routines are useful for future behavior anticipation, while this claim is evaluated only on synthetic Days 8--9. More importantly, each next-action question is constructed around a designated occurrence of a known ground-truth routine and asks the model to select the correct action from five routine-action candidates; the paper itself notes that this evaluates action selection at designated routine occurrences rather than prediction at arbitrary future timestamps. Therefore, the current results show that routine memory is useful for a controlled routine-centric prediction task, but do not yet establish its utility for anticipating natural future behavior. Even though EgoLife has no real Days 8--9, a natural alternative would be to construct memory from real Days 1--5 and evaluate on real Days 6--7. The shorter history would make routine induction harder, but this is precisely a realistic and informative setting. The absence of such a real temporal holdout is my main empirical concern.
2. The task taxonomy, related-work positioning, and baseline selection are incomplete for a problem that is fundamentally about pattern discovery. Both EgoRoutine-Bench and RMI are restricted to four predefined condition families: Place, Clock Time, Prior Action, and Frequency. The paper provides some motivation from the habit literature, but does not systematically justify why these four families are sufficient to represent natural routines or quantify how much of the naturally occurring routine structure in EgoLife they cover. More importantly, the experimental comparison focuses primarily on long-term memory agents, while the paper gives little attention to established work on association-rule mining, sequential pattern mining, temporal/periodic pattern mining, and spatio-temporal pattern mining. After RMI converts natural-language captions into structured occurrences with timestamps, places, and ordered contextual actions, these families admit natural statistical or symbolic baselines: sequential association rules for Prior Action, periodicity/inter-arrival modeling for Frequency, conditional place statistics for Place, and time-of-day clustering or histograms for Clock Time. The current experiments therefore establish that a pipeline specifically designed for routine discovery outperforms memory agents that were largely designed for query-conditioned retrieval, but do not show whether RMI is competitive with methods that are actually designed for recurring-pattern discovery. Relatedly, there remains a very large downstream gap between RMI memory and oracle routine memory. Although the paper provides a useful stage-wise failure analysis, it does not fully decompose how much of this downstream gap is caused by missing routine coverage, incorrect conditions, or interference from false-positive routines.
Questions for authors
See Weaknesses.
6 comments
Author Response
3.3. Comparison to natural statistical or symbolic baselines
Thank you for this suggestion. We implement the family-specific symbolic baseline that you outlined and test whether it can replace our LLM Condition Verifier. To remove action-discovery errors, we run it in the Given-Action setting on all 147 routines with the same target actions and extracted contexts as RMI, so the comparison isolates the induction step. For each action, the miner finds the most frequent place for PL, the densest 1–2h time window for CT, association rules over preceding actions for PA, and periodicity for FR, and selects the condition with the highest confidence and support. We apply the miner to the extracted contexts and to the consolidated evidence that our verifier receives.
Table 3.2: Comparison with statistical/symbolic pattern mining on Given-Action routine induction.
| Method (on given GT actions) | Consolidation | Pattern Induction / Verification | Given-Action Acc. (%) |
|---|---|---|---|
| Symbolic miner on extracted contexts | ✗ | Statistical / symbolic rules | 46.26 |
| Symbolic miner on consolidated evidence | ✓ | Statistical / symbolic rules | 58.50 |
| RMI (Ours, Qwen3.5-4B) | ✓ | LLM Condition Verifier | 76.42 |
Consolidation raises the miner from 46.26% to 58.50%, and the LLM verifier reaches 76.42% on the same evidence. The miner fails in two ways. First, string counting splits one preceding action into different phrasings, such as "pick up backpack" and "take backpack from cart." Consolidation merges these phrasings (Appendix B.3), which reduces this failure. Second, PL, CT, and PA actions occur about once a day, so the rule "once per day" gets the highest confidence and support and beats the true condition. The LLM verifier is robust to this failure, because it judges which condition the records support under the family definitions (Appendix B.4) instead of picking the highest count. We will add association-rule, sequential, periodic, and spatio-temporal pattern mining to the related work.
3.4. Decomposition of the downstream gap between RMI and Oracle memory
Thank you for raising this point. RMI reaches 50.34% next-action accuracy and Oracle memory reaches 94.78% (Table 3), a gap of 44.44 points. To split this gap, we apply three changes to RMI's memory, each on top of the previous one: (a) fix the conditions of the 9 routines with a correct action and a wrong condition, (b) remove the true FPs, whose action and condition match no ground-truth routine, and (c) add the ground-truth routines that RMI misses.
Table 3.3: Next-action accuracy after cumulative interventions on RMI's induced memory (Qwen3.5-4B).
| Memory | Accuracy |
|---|---|
| Original RMI | 50.34% |
| + (a) Correct the conditions of 9 routines | 56.46% |
| + (b) Remove true FPs | 56.69% |
| + (c) Add missing GT routines | 94.33% |
| Oracle memory | 94.78% |
Adding the missing routines raises accuracy from 56.69% to 94.33%. Fixing the 9 conditions raises it from 50.34% to 56.46%, and removing the true FPs raises it from 56.46% to 56.69%. The gap comes mostly from missing routines (37.64 of 44.44 points, 85%). Wrong conditions account for 6.12 points and FPs for 0.23 points.
The revised manuscript includes further clarifications and analyses informed by your feedback.
Look Forward to Your Feedback
Dear Reviewer Ba84,
We thank you for taking the time to provide constructive comments, which are greatly helpful in improving the quality of our paper. We have thoroughly addressed all the concerns raised, particularly those regarding downstream evaluation on held-out real caption histories, comparisons with statistical and symbolic pattern-mining baselines, and the analysis of the performance gap between RMI and oracle memory.
If you have any remaining concerns, feel free to let us know. We would be happy to continue the discussion if there are additional questions. Otherwise, we would be very grateful if you would consider our response and clarifications in your final evaluation.
We look forward to your feedback.
Best regards,
The Authors
Author Response
Thank you for your thoughtful assessment and insightful comments. We provide our responses to your concerns below.
3.1. More realistic evaluation of downstream utility on real future captions
Thank you for your suggestion. We induce routine memory from Days 1–5 and test next-action prediction on the real Days 6–7 EgoLife captions, both right before routine occurrences and at arbitrary timestamps. 142 of the 147 routines satisfy their conditions within Days 1–5, and we inject these 142 routines into the real Days 6–7 captions. We sample 142 positive timestamps right before an injected routine action and 1,000 negative timestamps at random points outside the injected occurrences. Each question adds an abstention option, "No applicable routine action," which is correct at negative timestamps.
Table 3.1: Next-action prediction on Days 6–7 with injected routine occurrences. Accuracy is measured at positive timestamps and FP rate at negative timestamps.
With Oracle memory, the agent answers 80.99% of positive timestamps correctly and triggers at 4.20% of negative timestamps (Table 3.1). Accurate routine memory therefore lets the agent recognize a condition that holds and reject one that does not. RMI's memory improves both: compared with WorldMM, RMI raises accuracy from 28.17% to 45.07% and lowers the FP rate from 33.50% to 22.40%. Accuracy at positive timestamps is lower than on the synthetic Days 8–9 in Table 3 for all memory inputs, for example from 94.78% to 80.99% with Oracle memory. This setting is harder: the memory comes from 5 days instead of 7, and each question adds the abstention option (Response 2.2 to Reviewer LLQP).
To rule out any effect of the injected occurrences, we also sample 1,000 random timestamps from the unedited Days 6–7 captions and count any routine action choice as a false alarm (Response 2.2 to Reviewer LLQP, Table 2.3). The trend holds: Oracle memory triggers at 4.10% of timestamps, and RMI still lowers the FP rate from WorldMM's 32.70% to 22.00%.
3.2. Justification of the proposed four routine types to cover natural routines
PL, CT, and PA follow the habit cues studied in psychology. Neal et al. (2012) report location, preceding actions, and time of day as habit triggers in daily life, and Keller et al. (2021) build habits around a set clock time or an existing routine. FR covers routines set by a schedule instead of a cue, such as taking medication at fixed intervals (Osterberg & Blaschke, 2005) or a throat lozenge every 2 hours (Table 1).
As a proof of concept, we checked whether the four families cover natural routines in EgoLife. We gave GPT-6-Astra the unedited Days 1–7 captions of persona A1 and asked it to list every repeated action and its condition, without naming any family. After human review removed 20% of the results, 48 activities remained. GPT-6-Astra then judged whether the condition of each activity fits the four families, and humans verified these judgments. 44 of the 48 activities fit. The other 4 (adjusting glasses, touching the face, stretching, walking) have no consistent condition at all, so no activity has a condition outside the four families. Of these 44, 10 recur with strong evidence, with PL (7) and PA (4) conditions; one fits both. No A1 activity has a frequency condition. This result comes from one persona, and it suggests that the four families cover most routine conditions in egocentric captions. We do not use these routines as ground truth because some have exceptions, and others can be guessed from common knowledge, such as washing hands at a sink.