acceptodds
Under review as a conference paper at ICLR 2027

NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

Abstract

We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do. Hindsight offers a valuable perspective on past decisions, although we often wish we could have simulated hindsight at the moment of choosing. If a system could generate plausible trajectories from one's personal history, such previews might help people formulate more realistic plans and make better informed decisions. We introduce NextMe-800, an approximately 800-hour first-person dataset from one volunteer over 126 days with 1 Hz images, gaze, and audio, captioned at five hierarchical abstraction levels from atomic actions to major activities. We formulate personalized action anticipation as open-vocabulary -step sequence prediction and construct NextAct, a 1,500-point benchmark combining NextMe-800 and EgoLife for within- and cross-person evaluation. Using an embedding-based soft edit distance as the metric, we evaluate how well different models can anticipate personal behavior and how prediction scales with context length. NextMe-800 and NextAct provide a months-long resource and evaluation framework for studying how far ahead personal behavior can be anticipated from egocentric observation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
egocentric video, personalized behavior prediction, action anticipation, longitudinal datasets, open-vocabulary evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e1dMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces NextMe-800, a longitudinal egocentric dataset containing approximately 800 hours of recordings from one individual over 126 days, with multimodal observations and five levels of hierarchical action captions. It also introduces NextAct, a 1,500-point action anticipation benchmark combining NextMe-800 and EgoLife. The paper validates action similarity metrics against human judgments and evaluates predictors across context lengths, abstraction levels, and prediction horizons.

Critical strengths
  1. The problem is clearly motivated, and the dataset design is well aligned with it. Following the same individual over several months provides a resource for studying personal routines and behavioral changes beyond the scope of short recordings.

  2. The dataset construction process is described in detail, including recording, privacy processing, dense captioning, and hierarchical annotation. This helps readers understand how the observations are transformed into a resource for behavior prediction.

Critical weaknesses
  1. It is not clear what the main contribution of the dataset construction is, beyond standard processing. Could you clarify which parts required substantive effort or verification, for example with a figure or discussion showing the main difficulties and how they were resolved?

  2. The paper motivates the dataset with helping people formulate more realistic plans, but it does not show why this requires months of recordings from a single individual. Beyond planning, what other applications would benefit from this dataset? And why do these applications need a long-term, single-person record rather than shorter or multi-person recordings?

Questions for authors

Please address the questions raised in Critical Weaknesses 1 and 2. These are clarification requests; the answers could affect my assessment.

Final Justification: The analysis of habit changes over 114 days makes a convincing case for recording the same person over several months. I also find the goal-based grouping rules and the audit of the underlying captions appropriate for constructing the benchmark. I would still welcome more direct validation of the higher-level event boundaries. Although I see limited methodological novelty in the annotation pipeline itself, the resulting benchmark enables a valuable study of how an individual’s routines evolve over time. I am therefore raising my rating from 2 to 3.

1 comment

Thank you for the clear questions; we will add the material below to the paper.

Q1. Which parts of dataset construction required substantive effort or verification?

Beyond standard processing, two parts required new design and validation.

1. An open-vocabulary metric. Measuring the similarity of actions described in natural language can be challenging. To avoid comparing free-text predictions with the ground truth, most prior work restricts actions to a few hundred verb–noun categories (e.g., Ego4D) or uses multiple-choice tests (e.g., EgoLife). Neither scales to training a digital twin with long-horizon, open-vocabulary rollouts. We therefore developed a more useful metric for anticipating these naturalistic behaviors. It supports both single-step and multi-step natural-language action prediction, and it aligns with human judgments (four annotators on 350 points). Table 2 compares how well each metric agrees with the annotators: the reranker (0.942) outperforms classical NLP metrics (0.692–0.827) and LLM judges (0.895–0.910).

2. Hierarchical prediction. Prior anticipation benchmarks predict at a single time scale of seconds (e.g., Ego4D LTA, EPIC-Kitchens). Fine-grained actions are informative but noisy, and small errors compound over long horizons, so a ten-step or day-long rollout becomes unreliable. To make predictions meaningful rather than predicting noise, our hierarchical representation also overcomes these challenges by merging events into five levels at goal shifts, not at fixed durations. Higher levels describe the same time span with fewer events and abstract away noise. Figure 8 in Appendix A shows one recording at all five levels. Predictions can also switch between levels to keep fine details where needed.

Q2. Why months of one person, and what other applications benefit?

Potential applications.

  1. Beyond planning, a key application is just-in-time adaptive interventions (JITAIs)—helping people achieve multi-month goals such as sticking to New Year's resolutions, keeping up with new papers, preparing an ICLR submission, or sustaining a 3-times-a-week workout routine. Because trial and error on real humans is costly, we need a human model to test whether different external interventions change the outcome, and select the most promising ones to apply.
  2. Behavior simulation in social science and psychology. Digital twins are usually built from questionnaires or interviews (e.g., Park et al., 2024), which may not fully represent how a person actually behaves. NextMe-800 allows building and testing a digital twin from four months of observed, naturalistic behavior.
  3. Memory for humans and agents, and habit formation and change, which need months of data. Individual patterns (e.g., a guitar-practice habit that was tied to one place and disappeared after the person moved, gym visits that depend on whether friends come along, or to-dos that are repeatedly scheduled but never finished) help predict later behavior. A new figure in Appendix K tracks 37 habits over 114 days: 8 formed, 5 faded, and 16 of 27 change points fall within a week of two life transitions.

Why months over shorter records: Slow habits and multi-month project cycles are hard to observe in short records, so ultra-long rollouts are what let us study the interplay among fading willpower, habit inertia, and external temptation across sparse L5 transitions (see habit-tracking examples in Appendix K). Quantitatively, one week yields only ~76 L5 events and the curve is still rising in Fig. 5b (EgoLife), whereas the 1,236 L5 events in NextMe-800 span enough history for the curve to level off.

Why one person over multi-person: Building a faithful personal digital twin requires longitudinal depth, whereas cross-individual generalization is separately evaluated on EgoLife's six participants (Appendix J).

Reviewer_rML4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NextMe-800, a longitudinal egocentric dataset containing approximately 800 hours of first-person recordings from a single individual over 126 days, including visual, gaze, and audio information. The recordings are processed into a five-level hierarchical caption structure, ranging from atomic actions to high-level activities, to support personalized behavior anticipation at different temporal granularities. The authors further propose NextAct, a 1,500-point benchmark combining NextMe-800 and EgoLife, which formulates personalized behavior anticipation as open-vocabulary K-step action sequence prediction. To evaluate predictions, they introduce an embedding-based soft edit distance metric and validate its agreement with human judgments. Experiments compare simple repetition baselines, frontier language models, and a retrieval-augmented agent. The results reveal that prediction performance generally decreases with longer horizons, intermediate abstraction levels tend to be more predictable, and increasing historical context does not consistently improve performance. These findings highlight the challenges of long-term personalized behavior anticipation and provide an evaluation framework for future research.

Critical strengths
  1. A valuable longitudinal egocentric dataset for personalized behavior anticipation. NextMe-800 provides approximately 800 hours of first-person recordings spanning 126 days, with visual, gaze, and audio information. Unlike existing datasets that primarily emphasize participant diversity or shorter recording periods, NextMe-800 enables the study of individual behavioral patterns and routines over several months. The five-level hierarchical annotation further supports behavior analysis at different temporal granularities. This represents a meaningful data contribution and provides a valuable resource for future research on personalized behavior anticipation.

  2. A systematic evaluation framework with meaningful empirical insights. The proposed NextAct benchmark extends action anticipation to open-vocabulary, multi-step behavior prediction. The authors also validate their embedding-based soft edit distance metric against human judgments, providing empirical support for its use in evaluating natural-language action sequences. Furthermore, the experiments investigate how prediction performance varies with historical context length, abstraction level, and prediction horizon. These analyses provide useful insights into the limitations of current language models and establish a foundation for future research in long-term behavior anticipation.

Critical weaknesses
  1. Limited generalizability of the longitudinal dataset. Although NextMe-800 provides an extensive recording of approximately 800 hours, the data comes from only a single individual. This raises concerns about whether the observed behavioral patterns and conclusions generalize across individuals with different lifestyles, routines, and environments. While NextAct incorporates additional participants from EgoLife, their recordings span a much shorter period and cannot fully validate the proposed months-long personalized anticipation setting. This limitation weakens the broader applicability of the dataset and the generalizability of the empirical findings.

  2. Concerns about the validity of the anticipation evaluation protocol. The benchmark provides models with the future target recording windows and evaluates the best of three generated trajectories. Although these choices help accommodate recording gaps and the stochastic nature of human behavior, they introduce information and oracle selection that would not ordinarily be available in realistic anticipation scenarios. Consequently, the reported scores may overestimate practical prediction performance. The paper lacks sufficient analysis of how these design choices affect the results, making it difficult to assess whether the benchmark reliably measures genuine future behavior anticipation.

Questions for authors
  1. Clarification of the future recording window information. The NextAct benchmark provides models with the target recording windows in advance. Could the authors clarify how much temporal information these windows reveal about future events? More importantly, how would prediction performance change if models were given only coarse temporal constraints rather than the exact recording windows? This would help determine whether the benchmark measures genuine future behavior anticipation or partially benefits from privileged future temporal information.

  2. Impact of Best-of-3 evaluation. The benchmark allows each model to generate three candidate trajectories and reports only the best-matching one. Could the authors additionally report single-trajectory performance and compare it with the Best-of-3 results? It would also be useful to examine whether the relative rankings of different models remain consistent under single-trajectory evaluation. This analysis would clarify how much the reported performance depends on oracle selection rather than the model's actual predictive reliability.

  3. Generalizability across individuals and temporal scales. Given that NextMe-800 contains recordings from only one individual, to what extent do the main empirical findings generalize to other participants? Could the authors provide separate evaluation results for NextMe-800 and EgoLife, particularly regarding the effects of historical context length and abstraction level? Such an analysis would help distinguish generalizable properties of personalized behavior anticipation from patterns specific to the primary participant.

1 comment

Thank you for these questions. They are valuable and deepened our thinking about what the benchmark measures.

Q1. How much do the target windows reveal, and what happens with coarse constraints?

The original target windows revealed when the K target actions start and end (including gaps where the camera was off). To test whether this timing helps, we compared three prompts that differ only in the window. Example for an L3 point with K=10:

  • Exact (original prompt): "predict 10 actions during Segment 1: 04-22 10:11:42 – 11:24:56, Segment 2: 04-22 14:12:10 – 14:33:26".
  • Coarse, rounded to a unit set by the level's mean event duration (half an hour for L3): "predict 10 actions during Segment 1: 04-22 10:00 – 11:30, Segment 2: 04-22 14:00 – 14:30".
  • No window: "predict the next 10 actions".

Coarse windows score the same as exact windows (see Fig. 11 in Appendix G). Without a window, ten-step scores drop, possibly because the model cannot estimate when the ten actions end and because they often span several recordings. One-step scores do not change significantly. More details are in Appendix G.

Q2. Single trajectory vs. best-of-3. Appendix H now reports both single-trajectory and best-of-3 scores for every model, separately for NextMe-800 and EgoLife.

  • Scores drop by 24–38% at K=1 and 12–19% at K=10.
  • Rankings stay relatively consistent: Kendall τ\tau = 0.90 (K=1) and 0.71 (K=10) on NextMe-800.

We keep best-of-3 as the main score because daily human behavior is a distribution, not a single most-likely point estimate.

Q3. Separate results for NextMe-800 and EgoLife. Figure 5 splits context scaling by dataset ((a) NextMe-800, (b) EgoLife) and Appendix C gives the per-person curves (Figure 10); Appendix J reports every model's scores per dataset and level (Table 14). The main difference is at L1: EgoLife rises and then declines, whereas NextMe-800 rises and plateaus. A large share of EgoLife L1 targets is spoken content, and with long histories the models shift from verbatim quotes (I say "...") to generic summaries (I continue / keep discussing ...), which lowers the score on speech targets rather than because information is lost. EgoLife's coarse-level curves also score much higher than NextMe-800's because L5 behaviors in EgoLife are more similar to each other than in NextMe-800, so the shared normalization baseline makes them look high; after re-normalizing with an EgoLife-specific baseline the trends are similar (Figure 6). Section 6 (p. 8) gives the detailed breakdown ("Why the two sources diverge" and "Why EgoLife L4–L5 look higher"). The abstraction level also behaves differently: NextMe-800 scores decrease from L1 to L5, whereas EgoLife's coarse levels score as high as or higher than its fine levels (Appendix J). We similarly update the human prediction baseline and report habit evolution and other interesting results in Appendices I and K.

Reviewer_HYdGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes NextMe-800, a dataset consisting of months-long recording of an individual, which is also used as a benchmark to validate leading VLMs on long-horizon future prediction.

Critical strengths
  • The dataset is built with careful considerations. I especially appreciate the effort the authors put in elaborating the design choices in Sec 3.

  • The experiments are well-designed with comprehensive results.

    • I especially like the collation of results in Tab 2.
    • Also, Line 325-326 that tackles local errors makes a lot of sense.
    • Line 352 which discusses predictor is given three chances also shows practical considerations.
    • Inclusion of Codex also provides an important agentic baseline.
    • Fig 5 might provide important insights for long-horizon memory managements for agentic systems.
Critical weaknesses
  1. Please discuss as NextMe-800 consists of only one individual, why wouldn't this introduce strong bias as one indivisual is unlikely a convincing representative of all possible human behaviors.

  2. In Tab 1, NextMe-800 is in fact not demonstrating the most competitive scale. Please discuss the significance of gaze and audio that other datasets missing.

  3. Fig 3, why not draw all five levels? This would be much more clearer.

  4. Is ultra long-horizon future prediction meaningful at all? Things can happen and people will change their minds, please discuss the practical functions of such study.

  5. Overall, experiments are not very surprising. An example is in Line 358, where short-horizon prediction needs only a few-minute context window, wheresas high-level events require longer context window. Please discuss.

  6. Tab 3 lacks a human baseline. It would be interesting to see how humans could perform.

Questions for authors

Please see the weaknesses section.

2 comments

Thank you for the positive assessment and the concrete suggestions.

W1. One individual introduces bias. One person's behavior indeed induces individual-specific patterns—the exact signal a digital twin is designed to capture. This kind of data collection faces an inherent trade-off between population breadth and longitudinal depth. Because continuous ultra-long personal recordings are rare, individual depth gives NextMe-800 unique value for benchmarking slow habit evolution and long-term consistency, where short-horizon multi-person datasets fall short. Nevertheless, to evaluate cross-individual generalization, we also merge a multi-subject dataset, EgoLife (six people, seven days), into the benchmark (Appendix J).

W2. Comparison to other datasets; significance of gaze and audio. NTCIR-18/LSC is longer in total hours, while NextMe-800 records 28× more densely with more behavioral detail. EgoLife records six people in a villa for one week, much like a reality show, so its behavior distribution differs from naturalistic life; NextMe-800 records one person's naturalistic life in the wild.

  • Audio. Transcribed speech is part of the captions, so predicting behavior includes predicting what the person says. Speech also reflects what the person is thinking; it can act as a think-aloud channel.
  • Gaze. Gaze is essential for captioning because a scene contains many objects (a desk, a computer, a phone), and only the gaze point shows which one the person is interacting with, e.g., which tab they click.

Both audio and gaze enrich our behavioral data and make the text captions more accurate.

W3. Fig. 3 with all five levels. This is a great suggestion. We added two new figures showing all five levels in Appendix A: one NextMe-800 example (Figure 8) and one EgoLife example (Figure 9, the same segment as Fig. 3).

W4. Is ultra-long-horizon prediction meaningful? One potential use is just-in-time adaptive interventions (JITAIs), which aim to help people reach real-life goals—such as New Year's resolutions, preparing an ICLR paper, or keeping up with exercise—goals that correspond to L5 events, which form and stabilize over weeks. Trial and error on real people is costly, so, as in robotics and autonomous driving, an accurate world model is essential. We need a person model, analogous to a model-based-RL world model, that predicts what will happen given past behavior, tests whether interventions change the outcome, and selects the most promising ones to apply. Beyond modeling the person (agent) alone, our ultra-long-horizon data also helps build a model of the real-world environment and its responses during interaction.

Crucially, short-horizon rollouts contain too few datapoints for sparse L5 transitions—such as whether a user keeps a 3-times-a-week workout routine or shifts their sleep schedule across weeks. People keep changing their minds, and ultra-long-horizon prediction is required to generate sufficient L5 trajectories to capture the dynamic interplay of fading willpower, habit inertia, and external temptation. Some interesting habit-tracking examples are in Appendix K.

W5. Context scaling. We agree that coarse levels needing more history is expected; the shape is less obvious. Both sources rise and then flatten, but they differ locally: EgoLife's L1 peaks early and declines, and its L5 catches up with the finer levels at long history, whereas NextMe-800 stays ordered (Figure 5). The L1 decline is dialogue turning into generic summaries, and EgoLife's coarse levels look higher mainly through the shared normalization baseline (Figure 6). Section 6 (p. 8) has details.

Saturation also differs by level: L1 and L2 stop improving after ~10 events, whereas L4 and L5 keep improving until ~300 and ~100, then stay flat to 70 days. Shankar & Howard (2013) show that a scale-invariant memory is optimal when a signal has no preferred time scale; our curves instead have a preferred scale that grows with abstraction, and Eagle & Pentland (2009) report the daily and weekly periodicity that could be the coarse scale.

W6. Human baseline. How well do people anticipate a specific individual's behavior from the same evidence the models receive? We conducted an additional experiment to answer this question. We found that third-person observers score close to DeepSeek-V4.1-Flash, whereas the data owner (the volunteer who recorded NextMe-800) scores on par with GPT-5.6-sol. Ironically, GPT may already predict the volunteer's next move as well as the volunteer does; perhaps it is time to trade our crystal balls for GPT. More details are in Appendix I.

References:

  • Shankar, K. H., & Howard, M. W. (2013). Optimally fuzzy temporal memory. Journal of Machine Learning Research, 14, 3753–3780.
  • Eagle, N., & Pentland, A. (2009). Eigenbehaviors: Identifying structure in routine. Behavioral Ecology and Sociobiology, 63(7), 1057–1066.

I appreciate the authros' response, which warrants an acceptance. I especially like the addition of Fig 8 and 9, which are very clear to read. Although I think it is still early to judge the practical usefulness of the ultra-long event prediction and the human baseline has shown that the leading models are already as good as humans, I believe this benchmark still serves as a good addtion to the community. Hence, I maintain my initial positive assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions in the updated PDF

We thank the reviewers. In the updated PDF, all changes from the submitted version are marked in blue.

We re-selected the NextAct evaluation points: many of the original points fell on the first event of a recording segment, so the new points are spread more evenly over the recordings.

We also re-ran all models on the new points, and updated every score, table and figure.

New analyses from the discussion: why the two datasets diverge under longer context (Section 6), habit formation and change (Appendix K).A third-person human baseline (Appendix I),per-person context scaling (Appendix C), the target-window ablation (Appendix G), single-trajectory scores (Appendix H), and results by dataset (Appendix J).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.