acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Planning: When and How Egocentric Signals Help Human Motion Forecasting

Abstract

Egocentric signals such as eye gaze and scene geometry are increasingly fed to human motion forecasting models, yet the reported gains are small and inconsistent across benchmarks. We ask when and why these signals help. On a controlled testbed spanning human–object (MoGaze), human–scene (GIMO) and human–human (EgoBody) interaction with two state-of-the-art forecasters, we train each network with and without egocentric input and study the gain. We find three answers. When: the gain peaks at input and output windows that cover the lag between gaze and body motion, which differs across datasets; a poorly chosen window eliminates the gain or makes it negative. Where: the gain concentrates in specific motion states and joints and is diluted or cancelled by aggregate metrics; on a pick-and-place benchmark the gain at the acting wrist is nearly twice the reported aggregate. What: current datasets under-supply the information egocentric signals need; oracle probes that provide the missing information substantially increase the gain; on EgoBody, a simple wearer status oracle raises the Global MPJPE gain from 4% to 14%. But what is missing is highly dataset-specific. Together, the three findings reframe a near-zero aggregate gain not as evidence that egocentric signals are useless, but as a consequence of poorly matched windows, indiscriminate aggregation and incomplete data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human motion forecasting, egocentric perception, multimodal learning, human-scene interaction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AuabOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important and under-examined question: why egocentric signals produce small and inconsistent improvements in human motion forecasting. Its main strength is the diagnostic perspective that modality value should be analyzed conditionally across temporal windows, motion states, body joints, and information availability.

The state- and joint-level analyses provide meaningful evidence that egocentric gains are heterogeneous and that aggregate metrics may conceal useful conditional effects. The controlled comparison between models with and without egocentric input is also preferable to comparisons that simultaneously change the input and architecture.

Nevertheless, several central claims are stronger than the experimental evidence supports.

Critical strengths
  1. The paper presents a clear and relevant motivation. Existing studies report small and inconsistent improvements from egocentric signals, but architecture changes and modality changes are often confounded. The controlled comparison between models with and without egocentric inputs is therefore useful.

  2. The paper is generally clearly structured. The when–where–what organization provides an intuitive framework for studying modality contribution. It encourages evaluation beyond a single aggregate performance number.

  3. The state- and joint-level analyses are informative. On MoGaze, the gain is larger during walking with fixation and at the acting wrist and elbow. These results support the claim that aggregate metrics can conceal heterogeneous effects.

  4. The paper distinguishes global trajectory prediction from local articulation using Global MPJPE, Local MPJPE, and Root Destination Error. This reveals that egocentric inputs mainly help global movement prediction rather than detailed local pose prediction.

Critical weaknesses
  1. Lack of technical novelty. The paper does not propose a new forecasting architecture, temporal alignment mechanism or intention representation. The technical contribution mainly consists of controlled ablations, window sweeps, subgroup analyses, and oracle probes using existing forecasters.

  2. Insufficient evidence for the gaze-to-motion lag claim: The paper claims that the gain peaks when the input and output windows cover the gaze-to-motion lag and the prediction-effectiveness lag. However, it does not directly measure either lag.

  3. Gaze and scene contributions are not separated: For C3 and C4, the egocentric input contains both gaze points and a scene point cloud. The reported gain therefore measures the joint contribution of gaze and scene geometry.

  4. The consistently small effect is primarily observed for Local MPJPE, which remains below 3% in all configurations. The evidence therefore supports the more specific conclusion that egocentric inputs provide small improvements in local articulation but highly variable improvements in global trajectory prediction.

  5. Paper writing, the presentation could also be improved: the oversized full-page table is visually overwhelming and contributes limited additional insight relative to its substantial space cost.

Questions for authors
  1. Have the authors directly measured the gaze-to-motion lag in each dataset? Why should the performance-maximizing window be interpreted as evidence of a behavioral lag?

  2. Can the authors isolate the effect of window length on GIMO using the same architecture, sampling rate, input modalities, and training protocol?

  3. Can the authors report separate pose-only, pose-plus-gaze, pose-plus-scene, and pose-plus-gaze-plus-scene results for C3 and C4?

  4. Are the state- and joint-specific effects statistically stable across subjects and random seeds? Can the authors provide confidence intervals and paired statistical tests?

  5. Since the EgoBody oracle is derived from future ground-truth motion, how do the authors distinguish information insufficiency from the expected benefit of supplying a compressed representation of the prediction target?

  6. Since no oracle reliably improves GIMO, what positive evidence supports the claim that its bottleneck lies in missing scene-navigation information rather than in model representation or optimization?

  7. Can the authors evaluate a simple state-conditioned, joint-specific, or lag-adaptive fusion method to verify that the diagnostic findings translate into improved and more stable forecasting?

  8. The authors suggest routing egocentric signals to specific motion states and joints. Could they implement a simple state-conditioned or joint-specific gating baseline to verify whether these analytical findings can translate into more stable improvements in forecasting performance?

Reviewer_kLEJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates the problem of 3D human motion forecasting, and analyzes when and where can eye gaze and scene geometry information help. The investigation is structured with 3 key questions:

  1. When the gain peaks with different input and output windows.
  2. Where the gain appears across different motion classes and joints.
  3. What information can improve the prediction across different datasets

They conduct experiments with three datasets (MoGaze, GIMO and EgoBody) and two forecasting architectures (GazeMotion and SIF3D) and show analysis and insights regarding each question they asked.

Critical strengths
  1. The paper's logic is clear and easy to follow. Although it does not follow the common structure used in research papers, the key idea and experimental results are shown clearly.
  2. The research question is well-motivated. Understanding whether the additional information helps with motion forecasting, to what extent and in which scenarios, is an important research direction.
Critical weaknesses
  1. As the paper is written more as an analysis rather than presenting a specific approach, the novelty of the method is ambiguous. The analysis provides valuable insights, but the paper does not introduce a fundamentally new analytical method.

  2. Finding 1 states that 'the gain peaks at input and output windows that cover the gaze-to-motion lag and the prediction-effectiveness lag respectively', but the actual gaze-to-motion lag distributions on these datasets are not measured. So the finding is mainly based on observation in Figure 2 but lacks solid evidence.

  3. As the 4 different settings in table 1 have different settings in the datasets, forecaster, input & output window size, rate and input modalities, and except from C1 & C2, the other settings all change more than one variable against the other settings. This makes it very difficult to compare across settings.

Questions for authors
  1. At L363, 'fixation when the eye-tracking confidence exceeds 0.7 and saccade otherwise'. How do fixation and saccade status defined and how are they related to the eye-tracking confidence? Is there any literature proving this correlation?

  2. How much is the variance of the window-dependent and state-specific gains with different seeds? Can you report the standard deviation over different runs?

  3. In section 6, in the pseudo stopspot paragraph for MoGaze, stopspot is derived from the future wrist position, in EgoBody future wearer state paragraph, oracle uses categorical labels computed from future ground-truth motion. As this information is related to the prediction target, how can you justify that the improvement is due to richer egocentric information instead of leakage of future motion GT?

Reviewer_T6WROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies when gaze and scene information improve 3D human motion forecasting beyond past body poses. It evaluates GazeMotion and SIF3D on MoGaze, GIMO, and EgoBody, comparing models with and without egocentric inputs. The experiments vary observation and prediction windows, break down gains by joint and motion state, and introduce oracle information. The authors attribute inconsistent gains to temporal misalignment, aggregation, and insufficient input information.

Critical strengths

The paper aims to address a useful question. Comparisons using existing predictors, together with separate global, local, and endpoint errors, help distinguish improvements in movement trajectory from improvements in body articulation.

Critical weaknesses
  1. The findings are largely expected, and the experiments do not establish the proposed explanations. The introduction already cites evidence that gaze precedes movement and mainly helps the root trajectory; per-joint error breakdowns are common practice in motion forecasting; and the core with/without-gaze comparison is a standard input ablation. The explanations offered beyond this are not established. The gaze-to-motion lag in Finding 1 is never measured, and the C2–C3 comparison changes the model, frame rate, and scene input along with the window. Each interaction type is represented by a single dataset, so effects attributed to interaction type cannot be separated from dataset differences. In Finding 2, fixation and saccade are defined by eye-tracking confidence (> 0.7), which reflects detection quality rather than eye-movement state.

  2. The oracle probes do not show that egocentric data lack the needed information. Both oracles contain future target information (the future wrist stopping point; future wearer-state labels), and Fig. 4 compares the augmented models with the no-ego baseline, without a model that receives the oracle but no gaze. The rise from 4.25% to 12.37% may therefore come from the future information itself. The failed probes on GIMO likewise do not show that its bottleneck lies in the data rather than the model.

Questions for authors

Needs further clarification

  1. How do pose-plus-oracle models without gaze perform on MoGaze and EgoBody, compared with pose-only, pose-plus-gaze, and pose-plus-gaze-plus-oracle models? What evidence distinguishes information missing from the inputs from information the models fail to extract?
  2. Can the gaze-to-motion and prediction-effectiveness lags be measured independently of the window sweep, and do they predict the optimal windows? For the C2–C3 comparison, how can window effects be separated from differences in model, frame rate, and scene input?
  3. In the SIF3D no-ego controls, is scene geometry removed together with gaze? Please report separate gaze-only and scene-only comparisons for C3 and C4. Fig. 2(c) is labeled "from gaze," although C3 also receives scene input.
  4. Why do Fig. 2 and Table 2 differ at the same windows (e.g., C1 Global gain 3.35% vs. 6.79%; C4 6.53% vs. 4.2%)? Does the five-seed averaging stated in Section 3.3 apply to every Fig. 2 cell?
  5. Does Section 5 use tracking confidence to label fixation and saccade, or does "confidence" refer to an eye-movement classifier?
  6. Which oracle probes, encoding strategies, and injection points were tested on GIMO, and what were their results? What supports attributing the remaining difficulty to the data rather than to the tested models?

Suggestions

  1. Fig. 2 is a low-resolution image and hard to read. Same problem for all figures. Table 1 and Sec. 3.2 disagree on C2's gaze input (direction vs. point). The text states that both baseline error bars in Fig. 4 cross zero, but the Global gain (4.25% ± 3.23) does not.
  2. I don't see how the future-work section could be relevant to the paper topic. It should be removed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.