acceptodds
Under review as a conference paper at ICLR 2027

CoordMotion: Learning Predictive Eye-Body Coordination States for Human Motion Forecasting

Abstract

Human motion forecasting is an important topic in the areas of computer vision, robotics, and human-aware artificial intelligence. Prior human behavioral studies have shown that eye–body coordination provides informative structure for understanding actions, intentions, and task progression. However, this relationship has not been explicitly explored for human motion forecasting. We present , a novel method that learns eye–body coordination as structured predictive behavioral states to forecast human motion. Specifically, (1) we learn structured eye–body coordination states that explicitly organize task-dependent eye–body action transitions from complementary semantic text and physical value; (2) we forecast the evolution of the learned coordination states and use the predicted future states to guide deterministic base prediction and horizon-dependent residual diffusion; and (3) we introduce a causal Coordination-Guided Verifier that leverages predicted coordination states to rank and select motion candidates, enabling coordination-aware test-time selection from multiple stochastic samples. We conduct extensive experiments on both deterministic and stochastic human motion forecasting tasks using the EE4D-Motion and Nymeria datasets. Experimental results show that our method outperforms state-of-the-art deterministic forecasting methods, reducing the mean per-joint position error by up to 9.7%, and produces higher-quality stochastic futures, reducing the average displacement error on EE4D-Motion by 20.2%. These results validate the significant potential of structured eye–body coordination states for human motion forecasting.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
human modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4a9nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This manuscript introduces CoordMotion, where the authors design it to learn, or say leverage, “eye-body coordination states” to predict 3D human motion. Specifically, it uses Qwen generated coordination textual descriptions (in the words of the authors “atomic action” or “action transition”) and per-signal motion statics of gaze/head/root/joints are fused, contrastively aligned, and split into four states (global/relational/phased/temporal), and distilled into a causal encoder. A GRU forecast future state that condition a deterministic predictor and 2-6 s residual diffusion model; a learned verifier keeps the top samples. As a result, the proposed CoordMotion achieves −9.7% J0–2, −20.2% expected ADE and −8.6% JPE vs. Ego3DLM, on EE4D-Motion and Nymeria.

Critical strengths

S1. The core idea of leveraging ‘eye-body coordination’ for human motion forecasting is well-motivated and intuitive. It is reasonable to explicitly leverage eye-body coordination as an explicit, forescastable intermediate stage. The setup of supervising model training offline by MLLM (qwen) generated text and statistics, then distilling so that no MLMM inference is needed at test time is also sensible.

S2. Clean observation of Improved performance. Within the same backbone it lowers EE4D J0–2 from 11.757 to 10.820 (Table 4a)

S3, Well-designed evaluation setup. The authors conduct experimental evaluation on two large egoentric benchmarks, across both deterministic and stochastic settings.

Critical weaknesses

Main: The most significant issue of the current manuscript is that the formulation and method are not understandable for the readers. For instance, at L#173–174: the inputs are used without a preliminary definition, such as ‘Given historical body motion B^h, eye E^h, visual features V^h, device trajectory T^h, and the current atomic action A_i, we construct the observed coordination states C_obs’ → It is nearly impossible for one to understand such formulation. So that the following methods are also blurred. To make the current manuscript understandable, major revision is needed. The distinction from prior gaze modeling usage is overstated. GazeMotion (Hu et al., 2024) already predicts gaze and learns gaze-pose relations; GazeMoDiff (Yan et al., 2023) uses gaze-motion graph attention. Further, highly related work, such as Learning Predictive Visuomotor Coordination (CVPR'26 Findings; joint gaze/head/upper-body forecasting on Ego-Exo4D) and GazeInterpreter (AAAI'26; LLM-generated eye–body-coordinated narrations on Nymeria) are not cited and discussed nor compared. Comparison validity and the verifier’s benefit remain uncertain. Nymeria uses a different split from published Ego3DLM (L#661-663), so Table 3 does not establish the advertised 8.6% gain. In Table 2, verification improves expected ADE2-6 (31.680 to 29.560 cm), but reduces APD (20.912 to 15.464) and worsens min ADE6s (24.239 to 24.776 cm). This trade-off does not independently establish improved physical or coordination validity.

—

Re:Clarity: Both Teaser (fig. 1) and Method (fig.2) figure are too dense, and hard to understand. Especially for the method figure, it is really difficult to follow and understand what’s the training/inference flow. A decomposed comparison would be beneficial. norm(·) in Eqs. 1, 3, 4, 7, 8 is undefined (L2? LayerNorm?) The Appendix F.3 is an incomplete writing Minor: L#177 cites Qwen2-VL but L#694 uses Qwen3.7-Plus L#290 "qquad" ? L#377 "CoodMotion" typo L#466 "Figure 4.3"?

Questions for authors
  • Tables 4, 6: With identical inputs, semantic supervision and parameter count, how does an unstructured latent planner compare with the four states?
  • There are also many scenarios that eye (gaze) and motion are not correlated at all. How would the proposed method behave or influence such scenarios?
  • Probably the thing puzzles me most is that: Why using text to carry the ‘eye-body coordination’ prior? There is neither reasoning in training nor inference. Why would text help? In the reviewers opinion, it might be hard or not optimal for textual modality to carry such fine-grained spatial prior.
  • Line#177: What is ‘evidence-based coordination’ ?
  • Line#184: ‘Semantic text capture coordination meaning’, here how the authors guarantee the text generated carries the ‘coordination meaning” rather than other noisy info, such as objects, environments, mood, etc?
9 comments
AuthorsOpenReview ↗

Thanks for your careful review! We sincerely appreciate that you recognize our well-motivated, intuitive core idea of modeling eye-body coordination for egocentric human motion forecasting.

I understand your thoughtful review consists of three parts: weaknesses, clarity‑related concerns, and questions. We respond to each part separately below.

W1. Manuscript formulation and method writing

Response. Thank you for this helpful comment. The revised formulation now makes the information flow and the roles of privileged versus deployable signals explicit. Following the reviewer's suggestion, we clarified the presentation by (i) defining the three stages and their interfaces, (ii) stating which signals are offline privileged supervision and which are available to the deployed model, (iii) defining L2 normalization, (iv) separating the four structured states, and (v) explaining candidate-specific alignment and causal verification.

Revision location. The overview was revised in lines 154-161 and 177-181; deployed inputs and privileged supervision in lines 184-200; physical coordination construction in lines 201-215; coordination-guided verification in lines 289-318.

W2. The distinction from prior gaze modeling was overstated

Response. Thank you for this important clarification. Prior work does model eye-body coupling; our specific contribution is to use predicted structured coordination states for full-body forecasting and causal verification. We have narrowed the wording accordingly: the revision explicitly discusses gaze-conditioned motion forecasting, predictive visuomotor coordination, and semantic eye-body interpretation, and limits our claim to the combination specific to CoordMotion.

Revision location. The corrected positioning is in lines 112-122 and 134-150, with the new references in lines 555-556 and 583-585.

W3. Nymeria comparison used a different split and did not support the advertised gain

Response. Thank you for identifying this protocol issue. The comparison is now evaluated under the released Ego3DLM protocol, removing the original split mismatch. When we prepared the original manuscript, Ego3DLM had not released its Nymeria recording split, sample mapping, or preprocessing code; these resources became available on October 20–21, 2026.

Following the reviewer's observation, we strengthened the comparison by reconstructing the benchmark using the released recording split and consecutive 5s-to-5s samples, retain the official Body23 topology and metric definitions, and report the aligned results in Table R3.

Table R3. Aligned Nymeria evaluation under the released Ego3DLM protocol. Errors are in millimeters; Diversity is higher-is-better.

MethodAPEJPEADE0-5sADE2sFDEFDE2sDiversity
Ego3DLM147.9364.5343.9205.9648.1312.61.0624
CoordMotion158.9343.2334.9191.2572.2283.41.1224

Under the fully aligned Ego3DLM protocol, CoordMotion improves six of the seven reported metrics. Relative to Ego3DLM, it reduces JPE from 364.5 to 343.2 mm, ADE over 0–5 s from 343.9 to 334.9 mm, ADE at 2 s from 205.9 to 191.2 mm, FDE from 648.1 to 572.2 mm , and FDE at 2 s from 312.6 to 283.4 mm. It also increases Diversity from 1.0624 to 1.1224, indicating that the accuracy gains do not come from collapsing the predicted motion distribution. Ego3DLM remains better on APE. We therefore report a broad but not uniform improvement: CoordMotion performs better on joint-level forecasting accuracy, displacement errors, final displacement errors, and diversity, while Ego3DLM retains the advantage on APE.

W4. The verifier's benefit remains uncertain

Response. Thank you for raising this important concern.

First, the verifier provides a causal improvement in expected retained-set quality, rather than a universal improvement in every oracle or diversity metric. The most important motivation of this module is: standard stochastic evaluation often reports a minimum error selected with future ground truth, which is not available in deployment. Our verifier instead scores generated candidates using predicted coordination consistency, physical plausibility, and temporal continuity. It reduces expected error by removing poor candidates, while accepting a small change in oracle coverage and a reduction in APD.

Second, when analyzing whether the results meet our design expectations: the causal benefit is shown in Table 2 of the revision: verification reduces ADE2-6 from 31.680 to 29.560 cm without future ground truth. On aligned Nymeria, it slightly improves JPE from 333.5 to 333.2 mm, although it does not improve every metric.

In conclusion, our verifier enhances the average quality of samples from multiple sampling without ground truth, which we consider the most important result for practical settings where ground truth cannot be used to select predictions.

Clarity 1. Figures 1 and 2 are too dense; training and inference flow are difficult to follow

Response. Thank you for this helpful suggestion. We have simplified Figure 2 by explicitly marking three types of flow: training-only supervision, the inference path, and shared data flow.

We explain this in text below: During training, synchronized gaze, action descriptions, visual observations, head trajectories, and body motion are used offline to construct semantic and physical coordination supervision. A causal student then learns to predict the coordination states using only historical body motion, head trajectory, and egocentric visual features. The predicted states guide the deterministic motion center and long-horizon residual diffusion.

Candidate sets generated by the diffusion model are subsequently used to train the Coordination-Guided Verifier with quality, physical-plausibility, and coordination-consistency objectives.

At inference, neither gaze, action descriptions, future motion, nor future ground truth is available. The student first predicts future coordination states from the observable history. These predicted states guide deterministic and stochastic motion forecasting, after which the verifier ranks and retains candidates using only the generated motions, historical memory, and predicted coordination states.

Revision location. The revised method overview is provided in Lines 154–161. The distinction between offline privileged supervision and deployed inputs is clarified in Lines 184–200. The coordination-guided prediction and causal verification flows are described in Lines 273–318.

Clarity 2. norm(.) is undefined

Response. Thank you for pointing out this missing definition. norm(.) denotes per-vector L2 normalization and is now explicitly defined as:

N2(x)=x/max⁡(∥x∥2,ϵ).N_2(x)=x/\max(\lVert x\rVert_2,\epsilon).

The same L2 normalization is used for the normalized semantic and physical representations in the appendix.

Revision location. The definition appears immediately after Eq. (1), in lines 197-200.

Clarity 3. Appendix F.3 is incomplete; the Qwen version is inconsistent

Response. Thank you for identifying these two issues. Both issues are resolved: Appendix F.3 is complete, and Qwen3.7-Plus is cited consistently. Following the reviewer's comments, we added the controlled ablation of the Global-only, Relational-only, Phase-only, Temporal-only, matched unstructured-latent, and complete four-state variants, together with matched settings and detailed analysis. We also corrected the model name.

Revision location. The completed structured four-state ablation, including the matched unstructured-latent baseline, is provided in Appendix F.3, lines 903-944. The main text names Qwen3.7-Plus in lines 191-200. The exact model and generation protocol are documented in lines 721-744, and the reference is listed in lines 638-639.

Clarity 4. Typographical errors (qquad, CoodMotion, and Figure 4.3)

Response. Thank you for carefully identifying these typographical errors. All reported typographical errors have been corrected. Specifically, we corrected the model name, the figure reference, and the raw typesetting token in Eq. (15), replacing qquadv with the properly formatted condition on the state index.

Revision location. The corrected Eq. (15) appears in lines 296-300.

Q1. Controlled unstructured latent planner versus the four structured states

Response. Thank you for requesting this controlled comparison. The matched comparison supports explicit four-state factorization over an equally capable unstructured latent. Following the reviewer's request, we add a strictly controlled baseline in which both planners receive identical body/history, head-trajectory, and visual inputs and use the same total teacher information, matched total latent capacity, four equally sized projection/FiLM branches, trainable-parameter count, optimizer, schedule, and evaluation manifest. For the unstructured baseline, we concatenate the Global, Relational, Phase, and Temporal teacher targets and apply a fixed dense orthogonal rotation. This preserves the information content and pairwise Euclidean distances while removing the explicit correspondence between latent blocks and coordination factors. The comparison therefore isolates semantic factorization rather than supervision, capacity, or parameter count.

Table R4. Controlled ablation of structured coordination states. Ja-b is mean MPJPE over the indicated forecast interval, in centimeters.

PlannerGlobalRelationalPhaseTemporalJ0-2J2-4J4-6
Global only✓---11.20828.73834.253
Relational only-✓--11.20729.88133.666
Phase only--✓-11.17629.64832.246
Temporal only---✓11.18128.98131.315
Unstructured latentMatched capacity; no semantic factorization12.34230.34040.384
Four structured states✓✓✓✓10.82027.06032.119

As shown in Table R4, every single state is useful but incomplete. The temporal state is strongest at the 4-6 s horizon (31.315 cm), consistent with its role in long-range evolution, while the global state is strongest among the single-state variants at 2-4 s (28.738 cm). Combining all four states gives the best performance at every interval. More importantly, the unstructured latent baseline is substantially worse despite matched information and capacity: the structured model improves J0-2 by 1.522 cm, J2-4 by 3.280 cm, and J4-6 by 8.265 cm. The widening long-horizon gap supports the claim that explicit factorization helps preserve distinct global, relational, phase, and temporal roles instead of allowing them to collapse into an undifferentiated condition.

Revision location. The single-state and full-model results are included in manuscript Table 6 and discussed in lines 464-477. The state definitions and independent projections are in lines 797-809. The full controlled comparison, including the matched unstructured-latent baseline, is now included in Appendix F.3, lines 903-944 (Table 7).

Q2. What happens when gaze and motion are not correlated?

Response. Thank you for this important question.

First, CoordMotion can represent gaze-body decoupling and does not require raw gaze at inference. It does not assume that gaze and body motion must be positively or synchronously correlated: low instantaneous correlation can still encode a meaningful temporal relation—for example, gaze may lead an upcoming movement or shift to the next target while the body completes the current action.

Table R5 (below, under Q5) further supports this observation. Under the coordination-perturbed setting, coordination-relevant terms (left/right, hand/arm vs leg/foot, gaze-leads/gaze-follows) are modified while synchronized physical cues, actions, objects and scenes remain fixed. This intentionally introduces conflicts between the described coordination relation and observed motion. Performance drops most prominently on gaze shift, eye/head balance and lead/follow probes—tasks directly tied to gaze-body coordination—demonstrating the representation is sensitive to consistency between semantic relations and physical evidence. In contrast, rephrasing or changing irrelevant nouns yields much milder degradation. Combined with validation results, this confirms CoordMotion does not enforce spurious gaze-motion alignment under inconsistency; it maintains decoupled relations or discards physically unsupported coordination.

And in our CoordMotion, gaze is used only as privileged teacher-side supervision. At inference, the causal student predicts coordination states from body history, head/device trajectory, and egocentric visual features. When gaze carries little motion-relevant information, the model can reduce reliance on the relational and temporal gaze cues and fall back on the available body, head, and visual evidence.

Reviewer_D8mjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoordMotion, a human motion forecasting framework that explicitly models eye–body coordination as structured predictive behavioral states. It learns global, relational, phased, and temporal coordination representations from semantic descriptions and physical motion features. The predicted coordination states guide deterministic motion prediction and stochastic residual diffusion, while a Coordination-Guided Verifier ranks and selects plausible motion candidates. Experiments on EE4D-Motion and Nymeria demonstrate improvements in both deterministic and stochastic motion forecasting.

Critical strengths
  1. The paper considers eye–body coordination in human motion forecasting and provides a reasonable motivation for explicitly modeling the relationships between gaze and body movements.
  2. The paper reports results on two datasets and provides ablation studies to examine the effects of different coordination representations and model components.
Critical weaknesses

1, Inconsistency Between Motion Visualizations and Video Content: In Figures 1 and 2, the illustrated human motions do not appear to align with the corresponding video frames. This raises concerns about the consistency of the visual examples and makes it difficult to interpret the relationship between the observed activities and the depicted human motions. The authors should clarify how these visualizations are generated and explain the apparent discrepancies.

2, Limited Methodological Novelty: The method mainly relies on standard Transformer and diffusion frameworks, with limited architectural innovation. It remains unclear how eye–body coordination is specifically leveraged beyond serving as additional conditioning information for motion forecasting.

3, Inconsistent Experimental Results: The experimental results do not consistently demonstrate the effectiveness of the proposed method, as it underperforms certain baselines on several metrics. In particular, Table 2 shows worse performance in terms of diversity (APD) and some minimum-error metrics, raising concerns about the overall benefits of the proposed approach.

4.Concerns About Qualitative Results: The visualizations in Figure 3 appear to show that the model without coordination produces more plausible motions that are closer to the ground truth than the full model. This raises questions about the effectiveness of the proposed coordination mechanism.

5, Incomplete Related Work: The review of human motion forecasting/prediction literature is not sufficiently comprehensive, with several relevant studies overlooked. The authors should expand the discussion of related work and better position their contributions within the existing literature.

Questions for authors

see the weakness

7 comments
AuthorsOpenReview ↗

Thank you for your thoughtful review and the time you dedicated to our work! Thank you for recognizing our motivation to explicitly model eye‑body relationships in human motion forecasting, as well as our experiments and ablation studies across two datasets.

Q1. Apparent inconsistency between motion visualizations and video frames

Response. Thank you for identifying this ambiguity. We have updated the schematic figures in the revised manuscript: the human figures now visualize the real predicted motions from our model, instead of abstract conceptual icons.

In the original version, the small human figures were only conceptual illustrations, not pose reconstructions synchronized with the video frames. In Figure 1(a), the three RGB frames remain the real Ego-Exo4D example showing the process from target anchoring to ball approach and coordinated execution. The updated figures in Figure 1(b) now show the actual predicted Body17 poses output by our model, corresponding to this example. Similarly, the figures in Figure 2 are replaced with real predicted motion samples generated by our model, to illustrate motion history, deterministic center, stochastic residual candidates, and verifier-ranked candidates.

Revision location. ** The revised captions of Figure 1 and Figure 2 clarify that the predicted motions are updated to match the video frames in panel (a).

Q2. Limited methodological novelty beyond standard Transformer and diffusion backbones

Response. Thank you for raising this concern. We agree that the Transformer and diffusion backbones are standard; this is intentional so that the contribution of eye-body coordination is not confounded with architectural scaling. The novelty is not a new generic backbone. It is the complete coordination-centered forecasting formulation: (i) learning explicit semantic and implicit physical coordination representations, (ii) predicting global, relational, phased, and temporal coordination states, (iii) injecting frame-aligned predicted states into deterministic and stochastic forecasting, and (iv) reusing the predicted states as a causal criterion for candidate verification. Coordination is therefore not simply concatenated as an extra input.

Under the same backbone, adding explicit and implicit coordination reduces deterministic J0-2 from 11.757 to 10.820 cm. For stochastic prediction, coordination reduces ADE2-6 from 33.894 to 31.680 cm, and coordination-guided verification further reduces it to 29.560 cm. The full set of structured states also outperforms every single-state variant. Because these improvements are obtained with controlled, standard backbones, they isolate the benefit of the learned behavioral representation.

Q3. Lower APD and worse minimum-error metrics in stochastic forecasting

Response. Thank you for this question.

First, the lower APD reflects an expected quality–coverage trade-off: the causal verifier improves retained-set quality without using future ground truth, rather than maximizing raw diversity. A larger APD does not necessarily imply better stochastic prediction, since implausible outliers also increase pairwise distance. This accuracy–diversity trade-off is common in human motion prediction. For example, CoMusion [1] improves ADE from 0.4310.431 to 0.3500.350 while APD decreases from 8.2668.266 to 7.6327.632; BeLFusion [2] similarly argues that raw APD must be evaluated jointly with behavioral consistency.

Second, for CoordMotion: Before verification, CoordMotion obtains expected errors of 28.842/34.915 cm over 2-4/4-6 s. The causal verifier reduces them to 27.060/32.119 cm, while the corresponding oracle minimum errors change only from 21.200/24.239 cm to 21.423/24.776 cm. Minimum error uses future ground truth to select a candidate and is unavailable at inference. Our verifier instead removes coordination-inconsistent or physically implausible samples using only predicted coordination states and observed history. The reduction in APD from 20.912 to 15.464 is therefore the intended quality-coverage trade-off, rather than evidence that useful multimodality has disappeared.

Revision location. The manuscript explicitly explains the expected-error, oracle-error, and APD trade-off in lines 416-431, lines 471-473.

Reference

[1] Jiarui Sun and Girish Chowdhary. CoMusion: Towards consistent stochastic human motion prediction via motion diffusion. In European Conference on Computer Vision (ECCV), pages 18–36, 2024.

[2] German Barquero, Sergio Escalera, and Cristina Palmero. BeLFusion: Latent diffusion for behavior-driven human motion prediction. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 2317–2327, 2023.

Q4. Qualitative examples appear to favor the coordination-free model

Response. Thank you for highlighting this visual ambiguity.

Genervation: The RGB frames and Body17 poses correspond to the same future timestamps but are not rendered from the same viewpoint. Specifically, the RGB frames are captured by the first-person Aria camera, whereas the 3D Body17 poses are translated relative to the last observed pelvis and rendered from a fixed canonical xx–zz viewpoint. Therefore, the skeletons are not expected to visually overlap with the person in the RGB images.

Each example is an 80-frame EE4D-Motion validation window sampled at 10 Hz, comprising 2 s of observed history and 6 s of future motion. For a window starting at ww, the displayed Body17 poses at future 2/4/6 s are taken at indices w+20+{19,39,59}.w + 20 + \{19, 39, 59\}. The corresponding 30 Hz Aria RGB frames are extracted at svideo+3(w+20+{19,39,59}),s_{\text{video}} + 3(w + 20 + \{19, 39, 59\}), where svideos_{\text{video}} denotes the sequence start in the aligned video. Gaze is retrieved at the same timestamps. We use the real validation sequences sfu_cooking028_2___7467___7869 (w=40w = 40), sfu_basketball_09_11___0___582 (w=60w = 60), and upenn_0728_Violin_2_2___0___6465 (w=200w = 200).

Analysis of visualization:

In the upper part, black is GT and orange is the without-coordination top-1 prediction. Blue is the CoordiMotion deterministic center over 0–2 s, and green is its verifier-selected stochastic continuation over 2–6 s. Thus, the blue-to-green change denotes the deterministic-to-stochastic transition within the full model, not two different methods.

The skeletons are pelvis-aligned using the same last-observation anchor, shown at identical timestamps, and rendered with common spatial limits. However, the Body17 poses are projected onto the xx–zz plane. Visual overlap in one panel is therefore not equivalent to lower MPJPE: depth differences are hidden, limb crossings obscure errors, and a method can appear closer in one projected frame while being less accurate over the full sequence.

The examples were selected from a fixed, evenly spaced pool of 48 validation windows per activity, requiring CoordiMotion to improve over the strict without-coordination top-1 result in both 0–2 s and 2–6 s. For Cooking, Basketball, and Violin, the 2–6 s MPJPE reductions are 14.453, 20.533, and 1.944 cm, respectively; the 0–2 s reductions are 0.093, 3.674, and 0.035 cm. Thus, the full model is closer to GT according to the evaluated sequence metric in all three shown cases, even where the overlay makes the baseline appear visually similar at an individual timestamp. For transparency, GT is used to select the displayed cases, but not to select a candidate within a case.

The curves below the poses are not positional-error curves. They compare normalized gaze dynamics and pelvis-relative body-response dynamics over time. Similarity of a response-dynamics curve therefore does not imply lower skeleton MPJPE.

The lower part has a different purpose: it shows all CoordiMotion stochastic endpoints under the same predicted coordination evolution. Candidates are ordered by the causal verifier score; green candidates are retained and gray candidates are rejected. GT is shown separately only as a reference and is not used for candidate ranking or retention. This panel illustrates coordination-consistent long-horizon multimodality, not 16 attempts to reproduce the single annotated future.

In conclusion, Figure 3 explains the mechanism: a deterministic prefix close to the observed continuation, multiple coordination-conditioned long-horizon futures, and causal proposal filtering.

Revision location. The revised Figure 3 and caption are at lines 454-458, with the full qualitative interpretation at lines 489-497. The aggregate ablation is at lines 389-397 and 464-477.

Q5. Incomplete related work

Response. Thank you for this helpful suggestion.

We have narrowed and clarified our novelty claim relative to the closest gaze and coordination studies. We expanded the related-work discussion and added two particularly relevant studies: Learning Predictive Visuomotor Coordination [1], which forecasts gaze, head, and upper-body coordination from egocentric inputs, and GazeInterpreter [2], which generates LLM-based eye-body-coordinated semantic narrations.

Our distinction is now stated more precisely. Prior methods use gaze as an auxiliary conditioning signal, forecast coordination as an end task, or generate retrospective semantic interpretations. CoordMotion instead learns eye-body coordination as predictive latent states, forecasts their evolution, uses them to guide future full-body motion, and uses them again for causal candidate verification.

Revision location. The expanded human-motion forecasting discussion is in lines 103-122. The new eye-body coordination comparison and the more precise positioning are in lines 126-150. The added references appear in the bibliography at lines 555-556 and 583-585.

Reference

[1] Wenqi Jia, Bolin Lai, Xu Cao, Miao Liu, Danfei Xu, and James M. Rehg. Learning predictive visuomotor coordination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3609–3619, 2026.

[2] Qing Chang and Zhiming Hu. GazeInterpreter: Parsing eye gaze to generate eye-body-coordinated narrations. In AAAI Conference on Artificial Intelligence, 2026.

Reviewer_fLQcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CoordMotion, an egocentric human motion forecasting framework that explicitly models eye-body coordination via predictive behavioral states. The method distills offline MLLM-generated coordination text into a causal planner, predicts future states, and conditions a hybrid deterministic-residual diffusion model followed by a post-hoc candidate verifier. Evaluated on EE4D-Motion and Nymeria, the proposed approach achieves lower displacement errors compared to existing direct-conditioning baselines.

Critical strengths
  • Explicitly modeling eye-body coordination dynamics introduces a behavioral inductive bias compared to naive direct conditioning.
  • Extensive experiments across EE4D-Motion and Nymeria demonstrate consistent reductions in trajectory and joint displacement errors.
Critical weaknesses
  • Pipeline Complexity & Scalability: The framework relies on an excessively complex multistage pipeline with heavy handcrafted inductive biases, which raises concerns regarding its scalability and architectural extensibility.
  • Limited Generalization & Robustness Evaluation: The empirical validation is restricted to within-dataset splits, lacking cross-domain transfer experiments. Furthermore, the paper does not evaluate real-world deployment robustness against practical sensor degradations, such as gaze tracking jitter, temporal desynchronization, or frame drops.
Questions for authors
  • Given that different benchmarks often adopt distinct joint topologies and capture setups, is there a feasible strategy to evaluate cross-dataset transfer (e.g., via a unified joint subset or skeleton retargeting if possible) to demonstrate out-of-domain generalization?
  • How does the model perform under realistic sensor corruptions, such as angular noise on gaze rays, tracking jitter, or intermittent dropped video frames?
5 comments
AuthorsOpenReview ↗

Thank you for your thoughtful review and the time you dedicated to our work! We appreciate your recognition of our contributions in explicitly modeling eye-body coordination dynamics introduces a behavioral inductive bias compared to naive direct conditioning. Regarding your concerns, we address each question individually, quoting them and providing our answer with additional experiments accordingly.

Q1. Cross-dataset transfer under different joint topologies and capture setups

Response. Thanks for your question! Following the reviewer's suggestion, we further strengthen the generalization evidence with two complementary out-of-domain protocols that separate genuine distribution shift from errors caused only by incompatible skeleton definitions. And CoordMotion transfers across datasets and remains more accurate than the motion-only baseline after topology harmonization.

  1. Full-topology direct transfer. We retain the complete forecasting representation and evaluate the frozen EE4D-trained model on Nymeria without target-domain training, fine-tuning, normalization fitting, or target-domain checkpoint selection. This is the more challenging end-to-end setting because it retains the combined shifts in skeleton recovery, capture system, subjects, activities, and sensors.
  2. Common-topology transfer. Following the observation of Rapczynski et al. [1] that incompatible joint definitions can create systematic cross-database errors, we deterministically project EE4D Body17 and Nymeria Body23 onto their shared anatomical topology, Common12: the bilateral shoulders, elbows, wrists, hips, knees, and ankles. The mapping uses only fixed semantic joint selection, reordering, and history-only coordinate canonicalization. It uses no target-domain optimization, target statistics, or future frames.

Table R1. Cross-dataset motion forecasting from EE4D to Nymeria. All models observe 2 s and forecast 2 s at 10 Hz. E->N uses the EE4D-selected checkpoint without target-domain training. Common12 denotes deterministic selection and reordering of the twelve shared joints.

Train->TestMethod[email protected] (cm)[email protected] (cm)[email protected] (cm)ADE0-2s (cm)
E->NLast Pose, Common1212.91823.63742.10023.885
E->NMotion-only, retargeting7.78515.77133.61517.106
E->NCoordMotion, without retargeting5.34313.22029.23413.643
E->NCoordMotion, retargeting6.31212.33227.87812.341

As shown in Table R1, the Last Pose baseline reveals the magnitude of the domain shift, with its error increasing from 12.918 cm at 0.5 s to 42.100 cm at 2.0 s. Although retargeting substantially improves the motion-only model, CoordMotion remains more accurate at every reported endpoint. Without retargeting, CoordMotion obtains the lowest 0.5 s error (5.343 cm), showing that the learned coordination representation transfers effectively in the near future. With retargeting, its advantage becomes clearer at longer horizons: CoordMotion reaches 12.332 cm at 1.0 s and 27.878 cm at 2.0 s, compared with 15.771 cm and 33.615 cm for the retargeted motion-only baseline. These results show that topology harmonization is useful, while the gains over the retargeted motion-only baseline cannot be explained by retargeting alone.

Revision location. The revised manuscript describes the two datasets and their different protocols in lines 353-359 and 400-404, reports within-dataset Nymeria results in lines 378-387, and documents the official Nymeria split and Body23 representation in lines 683-692.

AuthorsOpenReview ↗

Response to Reviewer fLQc (2/3) Q2: Missing or corrupted gaze and sensors

Q2. Missing or corrupted gaze and sensors

Response. Thanks for your insight comment. First, CoordMotion does not require gaze at inference and remains robust to the evaluated corruption of observable sensors. Gaze is privileged supervision used only for offline coordination-state construction, while the deployed forecasting model is gaze-free and predicts coordination states from observed body history, head trajectory, and egocentric visual features: C^=Sψ(Bh,Th,Vh).\hat{C}=S_\psi(B^h,T^h,V^h).

Therefore, missing gaze at deployment does not alter the inference graph.

Second, following the reviewer's suggestion, we add targeted robustness tests that separately consider (i) corruption of teacher-side gaze during offline supervision construction and (ii) corruption of observable deployment inputs.

Table R2. Robustness to corrupted observable inputs. Only observed inputs of frozen models are corrupted. J0-2 is mean Body17 MPJPE over the first two forecast seconds; FDE is measured at the last valid forecast frame. All errors are in centimeters.

Corrupted evidenceSettingUniEgoMotion J0-2CoordMotion J0-2CoordMotion FDE
NoneClean21.35910.82027.930
Ego DINOv230% frame dropout21.45311.32727.993
Head trajectory10% frame dropout23.34211.30027.959
Body skeletonRandom upper-body joint missing22.23414.50930.636
Body skeletonRandom lower-body joint missing21.53213.97330.256

As shown in Table R2, under 30% visual-frame dropout, UniEgoMotion J0-2 increases from 21.359 to 21.453 cm, whereas CoordMotion increases from 10.820 to 11.327 cm, with only a 0.063 cm increase in FDE. Under 10% head-trajectory dropout, UniEgoMotion increases to 23.342 cm, while CoordMotion increases to 11.300 cm, with a 0.029 cm increase in FDE. Missing body joints affects CoordMotion more strongly because body history is the primary forecasting signal. These results show that CoordMotion does not require test-time gaze and remains robust under the evaluated visual and head-trajectory corruptions, while degrading predictably when the central kinematic observation is incomplete.

References

[1] Michał Rapczyński, Philipp Werner, Sebastian Handrich, and Ayoub Al-Hamadi. A baseline for cross-database 3D human pose estimation. Sensors, 21(11):3769, 2021.

AuthorsOpenReview ↗

Response to Reviewer fLQc (3/3); Response to the Weaknesses

Thank you for your review! We address the two concerns below. The complete experimental settings, results, and Tables R1 and R2 are provided above in our detailed responses to Q1 and Q2, respectively.

Pipeline Complexity & Scalability

Response. Thank you for raising this concern. CoordMotion’s deployed pipeline is substantially simpler than its training-time supervision construction may suggest: gaze, action descriptions, and the language model are used only offline to construct coordination supervision and are not required at inference. At test time, the model predicts coordination states directly from observable body history, head/device trajectory, and egocentric visual features, uses them to guide motion forecasting, and ranks stochastic candidates with the causal verifier.

The framework uses standard Transformer and diffusion backbones rather than a specialized architecture. Its four coordination states are learned behavioral representations, not manually specified motion rules, and are incorporated through modular projection and conditioning branches. The same states improve both deterministic forecasting, reducing J0–2 from 11.757 to 10.820 cm, and stochastic forecasting, reducing ADE2–6 from 33.894 to 31.680 cm before verification and to 29.560 cm after verification. This shared interface allows the coordination representation to be incorporated into different forecasting backbones without redesigning the complete pipeline. We have also clarified the training-only supervision, inference path, and shared data flow directly in Figure 2.

Limited Generalization & Robustness Evaluation

Response. Thank you for raising this important concern. Following the reviewer’s suggestion, we strengthen the evaluation with cross-dataset transfer and targeted sensor-corruption experiments, reported in Tables R1 and R2.

As shown in Table R1, we evaluate a frozen EE4D-trained model directly on Nymeria without target-domain training, fine-tuning, target statistics, or future frames. After deterministic Common12 topology harmonization, CoordMotion achieves 12.332 cm at 1.0 s and 27.878 cm at 2.0 s, compared with 15.771 cm and 33.615 cm for the retargeted motion-only baseline. This shows that the gain persists under changes in subjects, activities, capture systems, sensors, and skeleton definitions.

As shown in Table R2, we additionally evaluate visual-frame dropout, head-trajectory dropout, and missing body joints. CoordMotion remains substantially more accurate than UniEgoMotion under every evaluated corruption. Under 30% visual-frame dropout, its J0–2 increases only from 10.820 to 11.327 cm; under 10% head-trajectory dropout, it reaches 11.300 cm. Missing body joints cause a larger and expected degradation because body history is the principal forecasting signal, but CoordMotion still outperforms UniEgoMotion.

Taken together, the supplementary experiments demonstrate the generalization and deployment robustness of our method.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal Discussion

Dear ACs and Reviewers,

Thank you very much for your valuable contributions to our work. We appreciate the reviewers’ thoughtful feedback and are encouraged by the positive overall assessment, including the recognition that explicitly modeling eye-body coordination dynamics introduces a behavioral inductive bias compared to naive direct conditioning, improved performance, and a well-designed evaluation setup.

During rebuttal, we substantially strengthened the paper through six targeted experiments (Tables R1-R6), a unified evaluation protocol, clearer presentation, and more precise claims. All manuscript revisions are marked in blue. To assist the AC and help streamline their review process, we provide a brief summary of the reviewer discussions below:

ReviewerMain concernOur revision and new evidence
fLQc: Weak AcceptCross-domain generalizationTable R1 evaluates frozen EE4D-to-Nymeria transfer without target-domain training and with Common12 topology harmonization. CoordMotion reaches 12.332/27.878 cm at 1/2 s versus 15.771/33.615 cm for the retargeted motion-only baseline.
Sensor robustnessTable R2 evaluates visual-frame dropout, head-trajectory dropout, and missing joints. CoordMotion remains more accurate than UniEgoMotion under every tested corruption.
Pipeline scalabilityWe clarified that gaze, text, and the language model are offline supervision only. Deployment uses observable history with standard Transformer/diffusion backbones, and Figure 2 now separates training-only, inference, and shared flows.
D8mj: Weak RejectVisual consistencyWe revised Figures 1-2 to show actual predictions and clarified Figure 3's synchronization, rendering, colors, and candidate-selection protocol.
Novelty beyond standard backbonesWe clarified that the contribution is the forecastable coordination representation: four structured states guide deterministic/stochastic forecasting and causal verification. Under the same backbone, coordination reduces J0-2 from 11.757 to 10.820 cm and ADE2-6 from 33.894 to 31.680 cm; verification further reduces it to 29.560 cm. We also expanded the closest related work and narrowed our positioning.
APD and oracle metricsWe clarified that minimum error requires future GT, whereas our verifier performs causal inference-time selection. We explained the APD quality-coverage trade-off and replaced the broad term “physical plausibility” with measurable kinematic consistency and retained-set quality.
4a9n: Weak RejectNymeria protocol and data consistencyTable R3 and revised Table 3 now use one released Ego3DLM protocol for all methods. Protocol-matched published results are retained; reproduced EgoLM/UniEgoMotion results are marked with a dagger; both CoordMotion variants use the same split and metrics. Tables 3/R3 now match exactly. CoordMotion improves six of seven metrics, including Diversity from 1.0624 to 1.1224, while Ego3DLM remains better on APE.
Structured versus unstructured statesTable R4 matches inputs, teacher information, capacity, branches, parameters, optimization, and samples. Structured states reduce J0-2/J2-4/J4-6 from 12.342/30.340/40.384 to 10.820/27.060/32.119 cm, isolating the benefit of explicit factorization.
Semantic groundingTable R5 separately perturbs coordination semantics and nuisance wording while holding physical evidence and training fixed. It supports representation grounding; we explicitly do not claim it establishes natural-decoupling robustness.
Verifier's physical validityTable R6 evaluates the same candidate sets using pose, bone stability, acceleration, and jerk. Retained candidates consistently outperform raw and rejected candidates, adding independent kinematic evidence beyond expected error.
Clarity and gaze-body decouplingWe rewrote the formulation, marked all flows in Figure 2, defined normalization, completed the appendix, standardized Qwen, expanded evidence validation, and corrected all noted errors. We clarified that masking and a learned gate avoid fixed gaze-body coupling and that gaze is absent at inference. We limit claims to the evaluated goal-directed activities; prolonged non-task decoupling remains future work.

Overall, the rebuttal improved the paper through broader evaluation, protocol consistency, better-controlled evidence, and more precise claims. Importantly, after reviewing our additional experiments, Reviewer fLQc stated: “My concerns are mostly addressed. I now still tend a positive rating.”

AuthorsOpenReview ↗

Summary of Rebuttal Discussion

These additions directly target the reviewers' concerns rather than changing the central method. The revised evidence supports our main conclusion that predictive eye-body coordination states improve human-motion forecasting under controlled comparisons, while the revised wording now distinguishes demonstrated forecasting and kinematic gains from broader robustness questions that remain outside the current benchmarks.

We respectfully hope the Area Chair will consider both the scope of the completed revision and the fact that the initially positive reviewer explicitly reported that the additional experiments mostly resolved their concerns and maintained a positive assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.