- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces NextMe-800, a longitudinal egocentric dataset containing approximately 800 hours of recordings from one individual over 126 days, with multimodal observations and five levels of hierarchical action captions. It also introduces NextAct, a 1,500-point action anticipation benchmark combining NextMe-800 and EgoLife. The paper validates action similarity metrics against human judgments and evaluates predictors across context lengths, abstraction levels, and prediction horizons.
Critical strengths
-
The problem is clearly motivated, and the dataset design is well aligned with it. Following the same individual over several months provides a resource for studying personal routines and behavioral changes beyond the scope of short recordings.
-
The dataset construction process is described in detail, including recording, privacy processing, dense captioning, and hierarchical annotation. This helps readers understand how the observations are transformed into a resource for behavior prediction.
Critical weaknesses
-
It is not clear what the main contribution of the dataset construction is, beyond standard processing. Could you clarify which parts required substantive effort or verification, for example with a figure or discussion showing the main difficulties and how they were resolved?
-
The paper motivates the dataset with helping people formulate more realistic plans, but it does not show why this requires months of recordings from a single individual. Beyond planning, what other applications would benefit from this dataset? And why do these applications need a long-term, single-person record rather than shorter or multi-person recordings?
Questions for authors
Please address the questions raised in Critical Weaknesses 1 and 2. These are clarification requests; the answers could affect my assessment.
Final Justification: The analysis of habit changes over 114 days makes a convincing case for recording the same person over several months. I also find the goal-based grouping rules and the audit of the underlying captions appropriate for constructing the benchmark. I would still welcome more direct validation of the higher-level event boundaries. Although I see limited methodological novelty in the annotation pipeline itself, the resulting benchmark enables a valuable study of how an individual’s routines evolve over time. I am therefore raising my rating from 2 to 3.
Thank you for the clear questions; we will add the material below to the paper.
Q1. Which parts of dataset construction required substantive effort or verification?
Beyond standard processing, two parts required new design and validation.
1. An open-vocabulary metric. Measuring the similarity of actions described in natural language can be challenging. To avoid comparing free-text predictions with the ground truth, most prior work restricts actions to a few hundred verb–noun categories (e.g., Ego4D) or uses multiple-choice tests (e.g., EgoLife). Neither scales to training a digital twin with long-horizon, open-vocabulary rollouts. We therefore developed a more useful metric for anticipating these naturalistic behaviors. It supports both single-step and multi-step natural-language action prediction, and it aligns with human judgments (four annotators on 350 points). Table 2 compares how well each metric agrees with the annotators: the reranker (0.942) outperforms classical NLP metrics (0.692–0.827) and LLM judges (0.895–0.910).
2. Hierarchical prediction. Prior anticipation benchmarks predict at a single time scale of seconds (e.g., Ego4D LTA, EPIC-Kitchens). Fine-grained actions are informative but noisy, and small errors compound over long horizons, so a ten-step or day-long rollout becomes unreliable. To make predictions meaningful rather than predicting noise, our hierarchical representation also overcomes these challenges by merging events into five levels at goal shifts, not at fixed durations. Higher levels describe the same time span with fewer events and abstract away noise. Figure 8 in Appendix A shows one recording at all five levels. Predictions can also switch between levels to keep fine details where needed.
Q2. Why months of one person, and what other applications benefit?
Potential applications.
Why months over shorter records: Slow habits and multi-month project cycles are hard to observe in short records, so ultra-long rollouts are what let us study the interplay among fading willpower, habit inertia, and external temptation across sparse L5 transitions (see habit-tracking examples in Appendix K). Quantitatively, one week yields only ~76 L5 events and the curve is still rising in Fig. 5b (EgoLife), whereas the 1,236 L5 events in NextMe-800 span enough history for the curve to level off.
Why one person over multi-person: Building a faithful personal digital twin requires longitudinal depth, whereas cross-individual generalization is separately evaluated on EgoLife's six participants (Appendix J).