acceptodds
Withdrawn from ICLR 2027

ReUP: Retrieval-Augmented Understanding and Prediction of Human Behaviour from Sparse Egocentric Signals

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An assistant such as smart glasses needs a holistic picture of its wearer. It must know what the wearer does, where the wearer is, how the body moves and what comes next. Existing systems run large models on continuous egocentric video or train one model per question. We show that two cheap signals suffice: the head pose the glasses already track and one image. We present ReUP, a Retrieval-augmented model for Understanding and Predicting human behaviour. Its memory holds fully instrumented walkthroughs, where every moment carries all the answers. Multi-modal contrastive learning gives its head-motion key rich semantics from the body and narration of each moment. Behaviour-guided contrastive learning draws together moments that share a place, an activity or a future. At query time, head motion and image each retrieve similar moments. For each task, a learned scorer picks the useful ones from both, and a light decoder turns them into an answer. The model blends what only each cue offers: the place from the image, the activity and body from the motion. With fewer than 2M parameters, one model recognises the current and next action, estimates and forecasts the body pose and localises the wearer. In unseen buildings and on an unseen dataset, it is on average 3.5% better than the strongest baseline of each metric. Retrieving recorded human behaviour gives a lightweight model a prior for holistic and effective understanding and prediction, from minimal input and in unseen scenes.

Primary area:
learning on time series and dynamical systems
Keywords:
Egocentric vision, Human understanding, Motion prediction, Memory retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jc7uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents REUP, a retrieval-augmented model that uses five seconds of head pose and one egocentric image to recognise current and future actions, estimate and forecast body pose, and localise the wearer. It learns a head-motion retrieval key using body and narration signals available during training, then retrieves fully annotated past walkthroughs and refines their answers with task-specific decoders. The paper evaluates the method in known buildings, unseen buildings and an unseen dataset.

Critical strengths
  • The research problem is clearly defined, and the retrieve-and-refine formulation is reasonable for sparse inputs.
  • The method gives the image and head trajectory complementary roles, and the input ablations support the value of combining them.
  • The paper evaluates five tasks under common input conditions and provides useful ablations of the encoder, memory and joint training.
Critical weaknesses
  • The practical problem differs substantially from the motivation. The vision of an everyday smart-glasses assistant is compelling, but the method assumes a memory whose entries already contain body pose, actions, future behaviour and, for localisation, position. Acquiring these annotations for a new environment may be more difficult than the inference problem the paper addresses.

  • The lightweight claim has an important qualification. The reported 1.92M parameters exclude the frozen CLIP image encoder, which dominates the stated computation. Replacing CLIP with a smaller encoder preserves much of the relative performance, but the paper does not establish whole-system cost on wearable hardware.

  • The scope of generalisation is limited. Behaviour prediction is evaluated in unseen buildings and one other dataset, whereas localisation requires recordings of the same building and is not evaluated in either unseen setting. The cross-dataset action results also rely on pseudo-labels.

  • The future-action evaluation needs clarification. Appendix A.1 suggests that Nymeria’s selected future narration often describes a later interval than the immediately following five-second body forecast, although both are presented as “next five seconds” predictions.

Questions for authors
  • Can the authors evaluate a memory built from realistically obtainable, noisy estimates rather than fully instrumented and annotated walkthroughs? What is the cost of adding a new building?

  • What are the complete parameter count, latency and memory footprint when the image encoder and retrieval database are included? Is there a measurement on wearable-class hardware?

  • How much does the behaviour-guided retrieval key improve over a simple nearest-neighbour system using the same memory, image encoder and decoder?

  • What exact time intervals do Nymeria’s future-action and future-pose targets cover relative to the input window?

Reviewer_Ya3qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReUP, a retrieval-augmented model that uses a five-second head-pose sequence and one egocentric image to understand and predict the wearer’s behavior. It indexes fully instrumented recordings containing body motion, actions, and location. A learned head-motion representation and the image retrieve similar moments; task-specific scorers and small decoders use them to estimate current and future actions, current and future body pose, and location. The paper evaluates the model in known buildings, unseen buildings, and an unseen dataset.

Critical strengths

Using one memory to support several tasks is a sensible approach when only head motion and a single image are available. The paper brings body tracking, forecasting, activity recognition, and localization into a shared setting. It also tests across buildings and datasets and includes ablations of the retrieval keys, memory, and joint training.

Critical weaknesses
  1. It is unclear how much retrieval accounts for the gains. The paper attributes its gains to retrieval, claiming that “the knowledge lives in the memory.” Removing the memory changes behaviour metrics by only 1.4–3.3% in unseen buildings (Fig. 5a), and retrieval alone is far worse than the full model (229.1 vs. 128.5 mm tracking APE, Table 1), showing that the learned corrections are important. Joint training improves the activity metrics by a few percent, but body metrics by less than 1% and localization by 2 cm (Table 5). However, the construction and training of the no-memory variant are not sufficiently specified to establish whether encoder pretraining, image predictors, supervision, and decoder capacity are matched. This makes the contribution of retrieval difficult to isolate.

  2. Memory collection and cross-wearer transfer need more testing. The query input is small, but the memory used in the experiments comes from recordings with motion capture, narration, and location annotations. Localization requires reference data from each target building. The paper describes adding a building by indexing a walkthrough, yet Setup A uses half of each evaluation building’s recordings as memory. The amount of recording time and spatial coverage needed is therefore unclear. The paper also does not explain whether stored bodies retain individual proportions or are retargeted to a common skeleton. It does not report whether participants overlap between memory and queries in Setup A, or how errors vary across wearer heights.

Questions for authors

Needs further clarification

  1. How was the no-memory variant in Fig. 5 constructed and trained? Does it retain the same encoder pretraining, image predictors, supervision, and decoder capacity? A matched retrieval-free control, including an end-to-end trained variant, would help isolate what memory contributes. For the competing baselines, how much of ReUP’s advantage remains with comparable image access and supervision, and how would a standard visual place-recognition baseline perform on localization? Please also report variation across training seeds for the key comparisons.
  2. Do the stored bodies retain participant-specific proportions, or are they normalized or retargeted to a common skeleton? Is Setup A participant-disjoint between memory and queries? Please explain how differences in wearer height and body proportions are handled; errors stratified by height differences between query wearers and retrieved examples would help assess cross-wearer transfer.
  3. Setup A uses half of each evaluation building’s recordings as memory, while the paper describes adding a building through a walkthrough. How much instrumented recording time and spatial coverage are needed per building, and how does localization perform with a single walkthrough? Please provide per-building results for the three evaluation buildings.
  4. Appendix A.1 states that future-action labels usually describe roughly [t0+10, t0+15) s, whereas body forecasting covers [t0+5, t0+10) s; Table 1 and Fig. 4 nevertheless describe both as the “next 5 s.” Please clarify the target intervals, report the distribution of actual target offsets, and assess whether aligning the action and body targets changes the results. What interval do ADT’s “next chunk” labels cover? A timeline would make these distinctions easier to follow.
  5. How are the 44 verb labels extracted from narrations, including the handling of synonyms such as “put” and “place”? How were the ADT labels produced from Gemini captions, and were any checked by humans? Since every prediction contains exactly four verbs while reference cardinality varies, please report label-count distributions and oracle F1 ceilings under the reported aggregation. A persistence baseline using predicted current activities, a head-motion-statistics baseline, and per-verb results would help establish how much action prediction goes beyond continuation and kinematic cues.
  6. Please clarify how ReUP differs from existing methods that jointly address multiple egocentric tasks, including Ego4o, EgoAgent, and the cited UniEgoMotion. The introduction’s broad contrast between ReUP’s holistic design and task-specific prior systems does not adequately distinguish it from these multi-task approaches.

Suggestions

  1. The four static qualitative windows make temporal smoothness, physical plausibility, and cross-wearer transfer difficult to assess; videos of representative successes and failures would help. Please also provide the re-implemented baseline code and the result files and table-regeneration script mentioned in the reproducibility statement, which I could not locate among the provided materials.
  2. Figure 1 does not make the task easy to understand. Although it separates database construction from querying, it does not make the motion-capture and narration requirements explicit or indicate the time interval corresponding to each output. A concrete query example with a timeline would help.
Reviewer_ybAMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReUP which uses a short head-pose trajectory and one egocentric image to estimate current activity, body motion, and location, and predict future activity and body motion. Its main idea is to retrieve similar moments from previously recorded, fully annotated walks (ie, building a database), then use small decoders to correct the retrieved answers. The model uses one retrieval database for all five tasks, with task-specific scoring and decoding.

Critical strengths
  • Personally, I like the idea of treating recorded experience as a "reusable source of supervision" at inference. Head motion is ambiguous therefore similar movements in the memory can provide plausible body poses and future trajectory predictions. The input image gives complementary context without requiring continuous video. This is a sensible application of retrieval to egocentric perception.

  • The results are competitive across several tasks, and the common input protocol makes the comparisons more useful. The memory-size, input, and separate-task ablations provide useful evidence beyond the full-system results.

Critical weaknesses
  • My main concern is whether retrieval adds enough to justify the dependence on an external memory. The model already has a head-motion encoder trained with rich supervision, image features, and task-specific prediction heads. Figure 5 has shown improvements compared to "no memory", but the improvements are fairly small in general (roughly 3%). A simpler model with the same encoder, supervision, and prediction capacity seems like a particularly important alternative. Otherwise, it is hard to tell whether the main benefit comes from retrieving past experience or from learning a good representation of head motion.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.