- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This manuscript introduces CoordMotion, where the authors design it to learn, or say leverage, “eye-body coordination states” to predict 3D human motion. Specifically, it uses Qwen generated coordination textual descriptions (in the words of the authors “atomic action” or “action transition”) and per-signal motion statics of gaze/head/root/joints are fused, contrastively aligned, and split into four states (global/relational/phased/temporal), and distilled into a causal encoder. A GRU forecast future state that condition a deterministic predictor and 2-6 s residual diffusion model; a learned verifier keeps the top samples. As a result, the proposed CoordMotion achieves −9.7% J0–2, −20.2% expected ADE and −8.6% JPE vs. Ego3DLM, on EE4D-Motion and Nymeria.
Critical strengths
S1. The core idea of leveraging ‘eye-body coordination’ for human motion forecasting is well-motivated and intuitive. It is reasonable to explicitly leverage eye-body coordination as an explicit, forescastable intermediate stage. The setup of supervising model training offline by MLLM (qwen) generated text and statistics, then distilling so that no MLMM inference is needed at test time is also sensible.
S2. Clean observation of Improved performance. Within the same backbone it lowers EE4D J0–2 from 11.757 to 10.820 (Table 4a)
S3, Well-designed evaluation setup. The authors conduct experimental evaluation on two large egoentric benchmarks, across both deterministic and stochastic settings.
Critical weaknesses
Main: The most significant issue of the current manuscript is that the formulation and method are not understandable for the readers. For instance, at L#173–174: the inputs are used without a preliminary definition, such as ‘Given historical body motion B^h, eye E^h, visual features V^h, device trajectory T^h, and the current atomic action A_i, we construct the observed coordination states C_obs’ → It is nearly impossible for one to understand such formulation. So that the following methods are also blurred. To make the current manuscript understandable, major revision is needed. The distinction from prior gaze modeling usage is overstated. GazeMotion (Hu et al., 2024) already predicts gaze and learns gaze-pose relations; GazeMoDiff (Yan et al., 2023) uses gaze-motion graph attention. Further, highly related work, such as Learning Predictive Visuomotor Coordination (CVPR'26 Findings; joint gaze/head/upper-body forecasting on Ego-Exo4D) and GazeInterpreter (AAAI'26; LLM-generated eye–body-coordinated narrations on Nymeria) are not cited and discussed nor compared. Comparison validity and the verifier’s benefit remain uncertain. Nymeria uses a different split from published Ego3DLM (L#661-663), so Table 3 does not establish the advertised 8.6% gain. In Table 2, verification improves expected ADE2-6 (31.680 to 29.560 cm), but reduces APD (20.912 to 15.464) and worsens min ADE6s (24.239 to 24.776 cm). This trade-off does not independently establish improved physical or coordination validity.
—
Re:Clarity: Both Teaser (fig. 1) and Method (fig.2) figure are too dense, and hard to understand. Especially for the method figure, it is really difficult to follow and understand what’s the training/inference flow. A decomposed comparison would be beneficial. norm(·) in Eqs. 1, 3, 4, 7, 8 is undefined (L2? LayerNorm?) The Appendix F.3 is an incomplete writing Minor: L#177 cites Qwen2-VL but L#694 uses Qwen3.7-Plus L#290 "qquad" ? L#377 "CoodMotion" typo L#466 "Figure 4.3"?
Questions for authors
- Tables 4, 6: With identical inputs, semantic supervision and parameter count, how does an unstructured latent planner compare with the four states?
- There are also many scenarios that eye (gaze) and motion are not correlated at all. How would the proposed method behave or influence such scenarios?
- Probably the thing puzzles me most is that: Why using text to carry the ‘eye-body coordination’ prior? There is neither reasoning in training nor inference. Why would text help? In the reviewers opinion, it might be hard or not optimal for textual modality to carry such fine-grained spatial prior.
- Line#177: What is ‘evidence-based coordination’ ?
- Line#184: ‘Semantic text capture coordination meaning’, here how the authors guarantee the text generated carries the ‘coordination meaning” rather than other noisy info, such as objects, environments, mood, etc?
9 comments
Clarity 1. Figures 1 and 2 are too dense; training and inference flow are difficult to follow
Response. Thank you for this helpful suggestion. We have simplified Figure 2 by explicitly marking three types of flow: training-only supervision, the inference path, and shared data flow.
We explain this in text below: During training, synchronized gaze, action descriptions, visual observations, head trajectories, and body motion are used offline to construct semantic and physical coordination supervision. A causal student then learns to predict the coordination states using only historical body motion, head trajectory, and egocentric visual features. The predicted states guide the deterministic motion center and long-horizon residual diffusion.
Candidate sets generated by the diffusion model are subsequently used to train the Coordination-Guided Verifier with quality, physical-plausibility, and coordination-consistency objectives.
At inference, neither gaze, action descriptions, future motion, nor future ground truth is available. The student first predicts future coordination states from the observable history. These predicted states guide deterministic and stochastic motion forecasting, after which the verifier ranks and retains candidates using only the generated motions, historical memory, and predicted coordination states.
Revision location. The revised method overview is provided in Lines 154–161. The distinction between offline privileged supervision and deployed inputs is clarified in Lines 184–200. The coordination-guided prediction and causal verification flows are described in Lines 273–318.
Clarity 2. norm(.) is undefined
Response. Thank you for pointing out this missing definition. norm(.) denotes per-vector L2 normalization and is now explicitly defined as:
The same L2 normalization is used for the normalized semantic and physical representations in the appendix.
Revision location. The definition appears immediately after Eq. (1), in lines 197-200.
Clarity 3. Appendix F.3 is incomplete; the Qwen version is inconsistent
Response. Thank you for identifying these two issues. Both issues are resolved: Appendix F.3 is complete, and Qwen3.7-Plus is cited consistently. Following the reviewer's comments, we added the controlled ablation of the Global-only, Relational-only, Phase-only, Temporal-only, matched unstructured-latent, and complete four-state variants, together with matched settings and detailed analysis. We also corrected the model name.
Revision location. The completed structured four-state ablation, including the matched unstructured-latent baseline, is provided in Appendix F.3, lines 903-944. The main text names Qwen3.7-Plus in lines 191-200. The exact model and generation protocol are documented in lines 721-744, and the reference is listed in lines 638-639.
Clarity 4. Typographical errors (qquad, CoodMotion, and Figure 4.3)
Response. Thank you for carefully identifying these typographical errors. All reported typographical errors have been corrected. Specifically, we corrected the model name, the figure reference, and the raw typesetting token in Eq. (15), replacing qquadv with the properly formatted condition on the state index.
Revision location. The corrected Eq. (15) appears in lines 296-300.
Q1. Controlled unstructured latent planner versus the four structured states
Response. Thank you for requesting this controlled comparison. The matched comparison supports explicit four-state factorization over an equally capable unstructured latent. Following the reviewer's request, we add a strictly controlled baseline in which both planners receive identical body/history, head-trajectory, and visual inputs and use the same total teacher information, matched total latent capacity, four equally sized projection/FiLM branches, trainable-parameter count, optimizer, schedule, and evaluation manifest. For the unstructured baseline, we concatenate the Global, Relational, Phase, and Temporal teacher targets and apply a fixed dense orthogonal rotation. This preserves the information content and pairwise Euclidean distances while removing the explicit correspondence between latent blocks and coordination factors. The comparison therefore isolates semantic factorization rather than supervision, capacity, or parameter count.
Table R4. Controlled ablation of structured coordination states. Ja-b is mean MPJPE over the indicated forecast interval, in centimeters.
| Planner | Global | Relational | Phase | Temporal | J0-2 | J2-4 | J4-6 |
|---|---|---|---|---|---|---|---|
| Global only | ✓ | - | - | - | 11.208 | 28.738 | 34.253 |
| Relational only | - | ✓ | - | - | 11.207 | 29.881 | 33.666 |
| Phase only | - | - | ✓ | - | 11.176 | 29.648 | 32.246 |
| Temporal only | - | - | - | ✓ | 11.181 | 28.981 | 31.315 |
| Unstructured latent | Matched capacity; no semantic factorization | 12.342 | 30.340 | 40.384 | |||
| Four structured states | ✓ | ✓ | ✓ | ✓ | 10.820 | 27.060 | 32.119 |
As shown in Table R4, every single state is useful but incomplete. The temporal state is strongest at the 4-6 s horizon (31.315 cm), consistent with its role in long-range evolution, while the global state is strongest among the single-state variants at 2-4 s (28.738 cm). Combining all four states gives the best performance at every interval. More importantly, the unstructured latent baseline is substantially worse despite matched information and capacity: the structured model improves J0-2 by 1.522 cm, J2-4 by 3.280 cm, and J4-6 by 8.265 cm. The widening long-horizon gap supports the claim that explicit factorization helps preserve distinct global, relational, phase, and temporal roles instead of allowing them to collapse into an undifferentiated condition.
Revision location. The single-state and full-model results are included in manuscript Table 6 and discussed in lines 464-477. The state definitions and independent projections are in lines 797-809. The full controlled comparison, including the matched unstructured-latent baseline, is now included in Appendix F.3, lines 903-944 (Table 7).
Q2. What happens when gaze and motion are not correlated?
Response. Thank you for this important question.
First, CoordMotion can represent gaze-body decoupling and does not require raw gaze at inference. It does not assume that gaze and body motion must be positively or synchronously correlated: low instantaneous correlation can still encode a meaningful temporal relation—for example, gaze may lead an upcoming movement or shift to the next target while the body completes the current action.
Table R5 (below, under Q5) further supports this observation. Under the coordination-perturbed setting, coordination-relevant terms (left/right, hand/arm vs leg/foot, gaze-leads/gaze-follows) are modified while synchronized physical cues, actions, objects and scenes remain fixed. This intentionally introduces conflicts between the described coordination relation and observed motion. Performance drops most prominently on gaze shift, eye/head balance and lead/follow probes—tasks directly tied to gaze-body coordination—demonstrating the representation is sensitive to consistency between semantic relations and physical evidence. In contrast, rephrasing or changing irrelevant nouns yields much milder degradation. Combined with validation results, this confirms CoordMotion does not enforce spurious gaze-motion alignment under inconsistency; it maintains decoupled relations or discards physically unsupported coordination.
And in our CoordMotion, gaze is used only as privileged teacher-side supervision. At inference, the causal student predicts coordination states from body history, head/device trajectory, and egocentric visual features. When gaze carries little motion-relevant information, the model can reduce reliance on the relational and temporal gaze cues and fall back on the available body, head, and visual evidence.
Thanks for your careful review! We sincerely appreciate that you recognize our well-motivated, intuitive core idea of modeling eye-body coordination for egocentric human motion forecasting.
I understand your thoughtful review consists of three parts: weaknesses, clarity‑related concerns, and questions. We respond to each part separately below.
W1. Manuscript formulation and method writing
Response. Thank you for this helpful comment. The revised formulation now makes the information flow and the roles of privileged versus deployable signals explicit. Following the reviewer's suggestion, we clarified the presentation by (i) defining the three stages and their interfaces, (ii) stating which signals are offline privileged supervision and which are available to the deployed model, (iii) defining L2 normalization, (iv) separating the four structured states, and (v) explaining candidate-specific alignment and causal verification.
Revision location. The overview was revised in lines 154-161 and 177-181; deployed inputs and privileged supervision in lines 184-200; physical coordination construction in lines 201-215; coordination-guided verification in lines 289-318.
W2. The distinction from prior gaze modeling was overstated
Response. Thank you for this important clarification. Prior work does model eye-body coupling; our specific contribution is to use predicted structured coordination states for full-body forecasting and causal verification. We have narrowed the wording accordingly: the revision explicitly discusses gaze-conditioned motion forecasting, predictive visuomotor coordination, and semantic eye-body interpretation, and limits our claim to the combination specific to CoordMotion.
Revision location. The corrected positioning is in lines 112-122 and 134-150, with the new references in lines 555-556 and 583-585.
W3. Nymeria comparison used a different split and did not support the advertised gain
Response. Thank you for identifying this protocol issue. The comparison is now evaluated under the released Ego3DLM protocol, removing the original split mismatch. When we prepared the original manuscript, Ego3DLM had not released its Nymeria recording split, sample mapping, or preprocessing code; these resources became available on October 20–21, 2026.
Following the reviewer's observation, we strengthened the comparison by reconstructing the benchmark using the released recording split and consecutive 5s-to-5s samples, retain the official Body23 topology and metric definitions, and report the aligned results in Table R3.
Table R3. Aligned Nymeria evaluation under the released Ego3DLM protocol. Errors are in millimeters; Diversity is higher-is-better.
Under the fully aligned Ego3DLM protocol, CoordMotion improves six of the seven reported metrics. Relative to Ego3DLM, it reduces JPE from 364.5 to 343.2 mm, ADE over 0–5 s from 343.9 to 334.9 mm, ADE at 2 s from 205.9 to 191.2 mm, FDE from 648.1 to 572.2 mm , and FDE at 2 s from 312.6 to 283.4 mm. It also increases Diversity from 1.0624 to 1.1224, indicating that the accuracy gains do not come from collapsing the predicted motion distribution. Ego3DLM remains better on APE. We therefore report a broad but not uniform improvement: CoordMotion performs better on joint-level forecasting accuracy, displacement errors, final displacement errors, and diversity, while Ego3DLM retains the advantage on APE.
W4. The verifier's benefit remains uncertain
Response. Thank you for raising this important concern.
First, the verifier provides a causal improvement in expected retained-set quality, rather than a universal improvement in every oracle or diversity metric. The most important motivation of this module is: standard stochastic evaluation often reports a minimum error selected with future ground truth, which is not available in deployment. Our verifier instead scores generated candidates using predicted coordination consistency, physical plausibility, and temporal continuity. It reduces expected error by removing poor candidates, while accepting a small change in oracle coverage and a reduction in APD.
Second, when analyzing whether the results meet our design expectations: the causal benefit is shown in Table 2 of the revision: verification reduces ADE2-6 from 31.680 to 29.560 cm without future ground truth. On aligned Nymeria, it slightly improves JPE from 333.5 to 333.2 mm, although it does not improve every metric.
In conclusion, our verifier enhances the average quality of samples from multiple sampling without ground truth, which we consider the most important result for practical settings where ground truth cannot be used to select predictions.