acceptodds
Under review as a conference paper at ICLR 2027

ProMac: Language-Guided World Action Modeling for Single-View Egocentric Navigation

Abstract

Egocentric visual navigation requires an agent to infer how to move through its surroundings toward a desired destination. We formulate this problem as predicting a feasible sequence of 2D waypoints from a single current egocentric image with a marked goal point. The goal specifies where to move, but reaching it requires reasoning about traversable space and intermediate motion while anticipating how the scene will evolve. This motivates representations that jointly capture semantic transition understanding and action-conditioned visual dynamics. We introduce ProMac, a world action model for egocentric navigation that uses an auxiliary chain-of-thought (CoT) reasoning reconstruction objective only at training time to guide world modeling. ProMac trains the visual token hidden states of a large multimodal model (LMM) backbone through two complementary pathways: (i) a frozen auxiliary language decoder provides supervision that compresses CoT reasoning information into the visual hidden states; (ii) a next-frame latent prediction head uses these “CoT-conditioned” visual patch hidden states to predict future visual token representations from a pretrained vision encoder. This language-guided world modeling objective grounds semantic transition reasoning in predicted visual dynamics, encouraging the trajectory policy to capture both how to move and how the scene will change under that movement. The reasoning and world modeling modules are used only during training. At inference, the vision-language model directly generates the 2D trajectory without explicit CoT generation or latent rollouts. On Ego4D Fut-Loc, ProMac reduces ADE, FDE, L2@First4, and DTW by 33.4%, 59.2%, 13.2%, and 49.1%, respectively, relative to trajectory-only Qwen3-VL-4B.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language-Action, Multimodal Language Models, Reasoning, Navigation, Egocentric, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_c6NtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ProMac predicts a 2D trajectory from a single egocentric image containing a red goal marker. A Qwen3-VL-4B backbone is trained with an adapter that maps final hidden states at visual positions into a frozen auxiliary language model to reconstruct structured navigation rationales. A separate linear head predicts frozen visual features of a future image from those same states. Training combines trajectory, rationale, and future-feature losses. Both auxiliary branches are removed at inference. The paper derives projected trajectory tasks with four-second horizons from AND and Ego4D Fut-Loc, selecting the final valid projected waypoint as the goal. ProMac reports ADE/FDE/L2@First4/DTW of 108.40/82.01/103.46/67.45 pixels on Fut-Loc and 16.83/9.04/23.39/14.42 on AND. Evaluation measures single-image, open-loop trajectory agreement, without closed-loop execution or safety validation.

Critical strengths
  • Richer supervision during training preserves direct trajectory generation at deployment. Removing both auxiliary branches avoids explicit rationale generation or latent rollouts at deployment. Relative to trajectory-only Qwen, Fut-Loc ADE decreases from 163.13 to 108.40 and DTW from 132.47 to 67.45, with corresponding improvements on AND. Table 3 shows that either auxiliary objective is useful independently, supporting the value of richer training supervision.

  • Dataset construction and supervision sources are documented in useful detail. The appendices describe projection, visibility filtering, endpoint backoff, recording-session or official splits, and restrictions to goal-only overlays as model inputs. Future images and complete trajectories shown to human annotators are identified as privileged annotation context, and validation human annotations are excluded from the training mixture. Discussion of temporal redundancy, limitations of annotation-consistency metrics, and the distinction between open-loop agreement and safe execution helps define the scope of the evidence.

Critical weaknesses
  • The action-conditioned world-modeling interpretation is not clearly supported by the actual prediction pathway.

The abstract emphasizes action-conditioned visual dynamics, but Eq. 4 and Appendix B describe a linear predictor from visual-position hidden states without explicit action conditioning. Section 3.3 states that it does not receive trajectory-token states, while Section 4.5 describes diagnostics conditioned on ground-truth intermediate trajectory tokens. The authors need to explain whether these tokens can influence the visual states being read. Under ordinary causal attention with image tokens preceding the answer, later trajectory tokens cannot affect earlier visual-position states. The described mechanism appears more directly consistent with goal-conditioned auxiliary future-feature regression than a demonstrated action-conditioned transition model.

The central combination benefit is also inconsistent. On AND, Full obtains ADE/FDE of 16.83/9.04, worse than reasoning-only at 16.09/7.57 and predictor-only at 16.13/7.43. On Fut-Loc, improvements over DriveVLA-W0 are only 109.91 to 108.40 ADE and 67.62 to 67.45 DTW. Without training-seed variation and recording-group-aware uncertainty, the additional contribution is difficult to assess. Negative nearest-later margins further indicate limited temporal discrimination in the future-state diagnostic.

  • Protocol discrepancies and missing simple controls weaken interpretation of the benchmark gains.

Section 4.8 and Table 4 describe history frames used in training and inference, whereas Appendix F describes adding history only at inference to frozen single-frame-trained checkpoints. The main table refers to 834 AND examples; Appendix F states that ADE uses 64 examples per dataset. The K=0 values exactly match the main table while being described as an archived reference. These protocols answer different questions, and it is unclear which produced the reported numbers. Consequently, the deterioration cannot currently be attributed to redundant frames.

The goal is directly supplied, and Appendix E finds 96.8% of AND image-space trajectories to be near-straight. Nevertheless, goal interpolation, a mean-start-to-goal path, and a lightweight goal-conditioned trajectory regressor are missing. ADE/FDE are computed after truncation to the shared prediction/target length, which may conceal prematurely terminated predictions; the reported FDE need not correspond to the actual target endpoint. Finally, Section 4.9 measures rationale-decoding latency even though deployment omits rationale decoding, so it does not establish end-to-end trajectory-generation efficiency.

Questions for authors

The following questions seek clarification and validation of the central conclusions, and the answers could change my rating.

  1. What conditions the future predictor? Please provide sequence order, attention masks, readout positions, and the checkpoint used for the diagnostic. How do ground-truth trajectory tokens in Section 4.5 influence the visual states in Eq. 4? If they do not, please revise the action-conditioning claims and compare explicit action-conditioned prediction against the unconditioned version.

  2. Which protocol produced the history results? Please reconcile Section 4.8, Table 4, and Appendix F regarding training, K values, sample counts, and reference models. Report history-trained models separately from inference-only history insertion using the same evaluation subset.

  3. Do the gains exceed simple geometric predictors and the closest method reliably? Could you add straight-line goal interpolation, a training-mean-start-to-goal path, and a lightweight goal-conditioned regressor, with results stratified by path curvature? Please report multiple seeds and recording-group-aware paired intervals for Full, individual branches, and DriveVLA-W0.

  4. How are output length and deployment cost handled? Please report invalid outputs, short trajectories, and length distributions, then reevaluate on a fixed temporal grid or penalize missing waypoints. Please provide trajectory-only inference latency, memory, and training cost under matched hardware, resolution, and decoding settings, and specify baseline training budgets.

Reviewer_MBceOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces goal-point-conditioned egocentric trajectory generation: given a single current egocentric image with the destination rendered as a red marker, predict the intermediate 2D waypoint path. The authors construct two benchmarks from AND (Project Aria) and Ego4D Future Locomotion by projecting future 3D ego-motion into the current view (9.4K/36.3K training examples), and propose PROMAC, a Qwen3-VL-4B trajectory policy trained with two training-only auxiliary objectives on the backbone's visual-token hidden states: (i) CoT reconstruction through an adapter into a frozen auxiliary LLM (§3.2), and (ii) a per-token linear head reconstructing frozen-encoder features of the future frame (§3.3). At inference both branches are removed and the model directly emits waypoints. PROMAC reports the best ADE/DTW on both benchmarks, and the authors additionally release 2K human-reviewed CoT rationales with an unusually careful consistency analysis (§4.7, App. A.5).

Critical strengths
  1. Well-engineered benchmark contribution with explicit leakage control. The dataset construction is documented to a degree rare for ICLR submissions: full projection equations for both pinhole and fisheye models (App. A.1–A.2), visibility filtering, goal-selection rules, an explicit audit that an early Fut-Loc artifact leaked the trajectory overlay into the input and was corrected before any training (App. A.6), and disclosure that checkpoint selection uses validation data (App. A.7). This is genuinely reproducible work.

  2. A clean and practically attractive inference recipe. Compressing reasoning into visual hidden states via a frozen auxiliary decoder, then discarding the branch at test time, gives trajectory-only inference with zero extra latency — a real deployment advantage over explicit-CoT VLAs such as AutoVLA, and the qualitative example in Fig. 8 (ADE 68.30 vs. 150.52 for vanilla Qwen) illustrates the gap concretely.

  3. Broad, same-backbone baseline coverage. Table 2 compares against fine-tuned trajectory-only and CoT Qwen3-VL-4B, AutoVLA, OneVL, LAW, and DriveVLA-W0 all re-trained on the same data/backbone, plus zero-shot 4B/8B/32B and closed-source frontier models. This is considerably fairer than the typical "our method vs. off-the-shelf numbers" comparison.

  4. Unusually honest negative and robustness analyses. The authors report that the full model does not uniformly beat single-objective ablations (Table 3), that the world-model head cannot reliably distinguish the supervised endpoint from nearby later frames (negative nearest-later margins of −22.7 and −9.6 ×10⁻³ MSE, §4.5), and that adding prior frames hurts (Table 4, K=1 worsens AND ADE 16.83→19.51). The human-annotation analysis (App. A.5) uses explicit independent-draw references (66.3%/94.5%) instead of presenting raw agreement as chance-corrected. This candor raises my trust in the numbers that remain.

  5. Structured-CoT format ablation. Table 11 isolates the effect of structured vs. natural-language rationales under matched training (18.8–21.7% error reduction on AND, ~1–2% on Fut-Loc), which is a useful datum for the CoT-supervision literature independent of the main method.

Critical weaknesses
  1. The headline improvement is measured against the weakest fine-tuned baseline; gains over the strongest baseline are within noise. The abstract's "reduces ADE/FDE/L2@First4/DTW by 33.4%/59.2%/13.2%/49.1%" is relative to trajectory-only Qwen3-VL-4B (163.13/200.91/119.26/132.47 on Fut-Loc, Table 2). Against the strongest baseline, DriveVLA-W0, PROMAC's Fut-Loc margins are 108.40 vs. 109.91 ADE (1.4%), 82.01 vs. 82.06 FDE (a tie), and 67.45 vs. 67.62 DTW (0.25%); on AND, PROMAC is worse on FDE (9.04 vs. 8.69). No seed variance or significance testing is reported anywhere, and App. E.4 acknowledges seed variation would be needed for exactly this kind of comparison. As presented, the central empirical claim — that adding CoT supervision to future-state prediction helps — is a ~1% unquantified-noise difference. The claim that PROMAC achieves "the lowest reported ADE and DTW" also omits that AutoVLA's L2@First4 on Fut-Loc (83.82) beats PROMAC (103.46) by 19%.

  2. The ablation does not support the claimed complementarity of the two auxiliary objectives (over-design risk). In Table 3, the Full model is worse than both single-objective variants on AND ADE (16.83 vs. 16.13 Predictor-only / 16.09 AuxLLM-only) and FDE (9.04 vs. 7.43/7.57), and worse than both on Fut-Loc L2@First4 (103.46 vs. 102.76/102.16). The authors themselves describe this as "a trade-off rather than uniform complementarity" (§4.4). Given this, the three-stage training pipeline (Stage 0 CoT compression, Stage 1 head calibration, Stage 2 joint) with its extra frozen 4B auxiliary LLM and future-frame encoder passes is not justified by evidence that the combination — the paper's titular contribution — is needed. A simpler "DriveVLA-W0 + one auxiliary loss" recipe appears to capture most of the benefit.

  3. The method is not action-conditioned, so "World Action Model" overstates what is built. App. B states the reconstruction head "introduces no attention layers or explicit action conditioning"; Eq. 4 predicts future features from current visual hidden states alone, without the trajectory or any action token as input. This is next-frame feature prediction, not an action-conditioned world model in the sense of LAW or DINO-WM, both of which the paper cites as action-conditioned. Consistently, §4.5 finds negative nearest-later margins — the learned predictor cannot discriminate the supervised future frame from frames 1–2 steps later — which directly undercuts the narrative that the model learns "how the scene will change under that movement" (abstract). The gap between the branding and the mechanism should be closed, either by conditioning on the trajectory or by renaming.

  4. No compute-matched or trivial-heuristic baselines. (a) PROMAC's training uses extra forward passes through a frozen auxiliary 4B LLM, a future-frame vision-encoder pass, and multi-stage optimization; no baseline is given equal training FLOPs/steps (e.g., trajectory-only Qwen trained longer), so part of the gain over the 163.13-ADE baseline may simply be more optimization. (b) The task renders the ground-truth final waypoint as the goal marker, and Table 12 shows 96.8% of AND validation targets are near-straight (R ≤ 1.05). A straight-line-to-goal-marker interpolation baseline is never reported; on AND it may be a nontrivial fraction of the achievable score, which would recalibrate how much of the 16.83 ADE is "navigation intelligence" vs. "connect to the dot." Relatedly, FDE is nearly free in this task design (the endpoint is given in the input), yet its 59.2% reduction is in the abstract.

  5. The efficiency section (§4.9) is vacuous as evidence. The reported gains — 0.98% on Fut-Loc and 3.87% on AND "paired CoT-generation latency" — compare rationale-generation latency of PROMAC's auxiliary readout against vanilla Qwen CoT. But PROMAC's selling point is that it generates no CoT at inference; the deployment-relevant comparison (trajectory-only latency vs. trajectory-only Qwen) is definitionally identical and is not what is measured. Reporting sub-4% differences on a comparison that the method's own design makes irrelevant reads as metric dressing and should be cut or replaced with end-to-end inference latency including the trajectory-only baseline.

  6. The "2K human-annotated CoT" contribution is oversold relative to what the model trains on. Only 907 Aria and 894 Fut-Loc training samples have usable human revisions (App. A.5); the rest of the rationale supervision is model-generated (per the AI Use Statement), and App. A.5 explicitly states no human-vs.-generated CoT training ablation is reported. The human annotation effort is real and well-analyzed, but its causal contribution to the trajectory results is currently zero-evidence — it may be the human review that matters, or raw template coverage.

  7. LLM-judge evidence is framed as supportive when it is at best neutral. In §4.6/Fig. 4, the CoT decoded from PROMAC's visual states scores below the vanilla Qwen reference on overall (3.53 vs. 3.60) and grounding (3.33 vs. 3.43), and the judge is Qwen3-VL-235B — the same model family as both generators, inviting self-preference bias. The text converts this into "strongly indicates that the compressed CoT related information... [is] beneficial for improved trajectory generation," which conflates "reasoning text is recoverable from the states" with "the recovery causes the trajectory gains." The latter is only supported by the Table 3 ablation, which (per Weakness 2) is mixed.

  8. Evaluation is validation-only, on few sessions, with shared-prefix truncation. Checkpoint selection uses validation performance on both benchmarks (App. A.7), AND validation covers only 3 recording sessions, and ADE/FDE/L2@First4 truncate predictions and references to their shared length K (§4.2), which can flatter models that systematically over- or under-generate waypoints (Fig. 8 shows 39–40 predicted vs. 33 reference waypoints). NaviTrace (Windecker et al., 2026), the closest single-image 2D-trace method discussed in §2.1, is never benchmarked.

Questions for authors

SEE WEAKNESS

Reviewer_ibYJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProMac, a language-guided world action model for single-view egocentric navigation. The method combines chain-of-thought (CoT) reasoning supervision with future visual feature prediction to improve trajectory generation from a single goal-marked image. Specifically, a frozen language decoder provides auxiliary reasoning supervision, while a future-state prediction head encourages the model to learn action-conditioned visual dynamics. Both components are used only during training, avoiding additional CoT generation or world-model rollout overhead during inference. Experiments on AND and Ego4D Fut-Loc demonstrate competitive performance against several baselines.

Critical strengths
  1. Well-motivated Design: The method combines language reasoning supervision with future visual feature prediction while avoiding additional CoT generation overhead during inference. This provides a practical approach to incorporating reasoning and world modeling into egocentric navigation.

  2. Strong Experimental Results: The method achieves competitive performance on both AND and Ego4D Fut-Loc, with comparisons against a diverse set of baselines. The experimental results demonstrate the effectiveness of the proposed training framework for trajectory prediction.

Critical weaknesses
  1. Insufficient Evidence of Complementarity: The ablation results show that combining CoT supervision with future-state prediction does not consistently outperform either component alone. This raises concerns about whether the two components provide complementary benefits, which is a central motivation of the proposed method. Additional analysis is needed to clarify why their combination is beneficial and under what conditions the two objectives reinforce each other.

  2. Insufficient Validation of World Modeling: The future visual feature prediction results reveal limited temporal discrimination between nearby states, raising concerns about whether the model learns meaningful action-conditioned dynamics. Since world modeling is a central contribution of the paper, the current evidence is insufficient to establish that the predicted representations capture meaningful temporal transitions rather than general visual similarity.

  3. Limited Methodological Novelty: The method primarily combines existing CoT supervision and future-state prediction techniques, and its novelty over related approaches such as OneVL remains unclear. While the overall design is reasonable, the paper does not sufficiently establish which methodological contributions go beyond existing approaches.

Questions for authors
  1. Complementarity of the Two Objectives: The ablation results suggest that combining CoT supervision with future-state prediction does not consistently improve performance over either component alone. Could the authors clarify why the two objectives are considered complementary and explain the cases where their combination provides limited or negative improvements?

  2. Effectiveness of World Modeling: The future visual feature prediction results show limited discrimination between temporally adjacent states. Could the authors provide additional evidence that the learned representations capture meaningful action-conditioned dynamics, rather than primarily reflecting visual similarity between frames?

  3. Novelty Compared with Related Work: Could the authors clarify the key methodological differences between ProMac and existing approaches such as OneVL? In particular, which components constitute the main technical novelty beyond combining reasoning supervision and future-state prediction?

Reviewer_reCNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ProMac, a language-guided framework for single-view egocentric trajectory generation that incorporates chain-of-thought (CoT) supervision and future visual-state prediction during training while maintaining trajectory-only inference. The problem is relevant, and the experimental results demonstrate improvements over trajectory-only fine-tuning. However, I have several major concerns regarding the methodological novelty, technical formulation, and experimental validation

Critical strengths
  1. The paper addresses a relevant problem in egocentric navigation, and the idea of using reasoning and future-state supervision only during training is practically appealing.

  2. The proposed framework is relatively straightforward and does not require additional reasoning generation or world-model rollouts during inference.

Critical weaknesses
  1. Limited methodological novelty. The main approach combines CoT supervision and future visual-feature prediction, both of which have been explored in prior work. The paper does not clearly establish what is fundamentally new compared with methods such as OneVL and DriveVLA-W0.

  2. Insufficient evaluation to support the main claims. The shared-prefix ADE/FDE evaluation may overlook incomplete trajectories and failure to reach the goal. Moreover, the improvements over DriveVLA-W0 are marginal on Fut-Loc, without multi-seed results or statistical significance analysis. The evaluation is also based on validation sets rather than independent held-out test sets.

Questions for authors
  1. What is the key methodological difference between ProMac and existing approaches such as OneVL and DriveVLA-W0, beyond combining auxiliary supervision on shared visual representations?

  2. How does the future-state predictor capture action-dependent dynamics without explicitly receiving action or trajectory inputs? Can the authors provide evidence that it learns meaningful state transitions rather than general future-feature correlations?

  3. Why does combining the two auxiliary objectives hurt ADE and FDE on AND? Have the authors investigated possible conflicts between the objectives or sensitivity to their loss weights?

  4. Can the authors report goal-reaching success rates, trajectory completion rates, and multi-seed results? These would help establish whether the reported improvements translate into more reliable navigation performance.

Comments on the paper

1 comment
Area_Chair_XWWnOpenReview ↗

Please check authors' rebuttal

Dear reviewers,

As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.