acceptodds
Under review as a conference paper at ICLR 2027

EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation

Abstract

Autonomous endoscopic navigation remains challenging due to tissue deformation, transient occlusions, and rapid viewpoint changes. Existing policies typically predict actions from current observations without explicitly modeling future anatomical dynamics, limiting robustness in safety-critical settings. We present EndoWAM, to our knowledge the first World Action Model (WAM) for generalizable robotic endoscopic navigation. EndoWAM introduces future grounding, which predicts task-relevant target regions in future observations from intermediate denoising features of a video world model. A lightweight diffusion transformer for future target prediction is coupled with a discrete action expert through a shared predictive representation, enabling target-aware dynamics modeling and real-time control with a single denoising pass. We further introduce EndoMotion, a robotic endoscopic motion dataset covering ureteroscopy, esophagoscopy, and endoscopic retrograde cholangiopancreatography (ERCP). EndoWAM consistently outperforms strong baselines and alternative grounding strategies, while exhibiting zero-shot generalization to unseen viewpoints, environments, and targets. These results demonstrate the effectiveness of predictive, target-grounded world-action modeling for robust and generalizable navigation in deformable endoscopic environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Robotic Endoscopy, Autonomous Navigation, Future Grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_utW9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EndoWAM, a world-action model for generalizable robotic endoscopic navigation. The proposed framework leverages a pretrained video world model to extract predictive representations from a single denoising pass, which are used to condition a discrete action expert and an auxiliary future-grounding module. The grounding module reconstructs latent representations of target regions in future observations during training and is removed at inference to improve efficiency. The authors also introduce EndoMotion, a multi-procedure robotic endoscopy dataset with target annotations and viewpoint augmentation. Closed-loop experiments on physical anatomical phantoms demonstrate substantial improvements over existing imitation-learning and vision-language-action baselines, as well as alternative grounding strategies.

Critical strengths

The paper provides extensive closed-loop evaluations on a physical robotic endoscope across three anatomical phantom settings, including comparisons with established baselines, alternative grounding strategies, and ablation studies. The consistent performance improvements provide meaningful empirical support for the effectiveness of the proposed approach in robotic endoscopic navigation.

Critical weaknesses
  1. The action labels are generated from tracked target observations using a hand-designed centering and advancement rule. However, the paper does not evaluate this rule combined with the same fine-tuned Grounding DINO in closed-loop control. Without this direct visual-servoing baseline, it remains unclear how much the learned policy improves upon its rule-based supervision.
  2. The contributions of future grounding and predictive dynamics modeling are not sufficiently isolated. A matched current-frame target reconstruction baseline is missing, making it difficult to establish the benefit of future grounding specifically. The video prediction loss is also not ablated, leaving its independent contribution to the reported performance improvements unclear.
Questions for authors
  1. How does a direct visual-servoing controller using the same fine-tuned Grounding DINO and the action rules described in Appendix B perform in closed-loop evaluation? This comparison would help clarify the benefit of the learned policy beyond its rule-based action supervision.
  2. How does the proposed method perform when the Grounding DiT reconstructs current-frame target regions instead of future target regions? A controlled comparison would help establish whether the improvement comes specifically from future grounding rather than target-region reconstruction in general.
  3. What is the contribution of the video prediction objective to navigation performance? Could the authors provide an ablation without this objective while retaining the same backbone and grounding supervision to distinguish the benefit of predictive dynamics learning from that of pretrained visual representations?
  4. In Eq. 1, future visual dynamics are modeled without explicitly conditioning on robot actions, although the executed actions directly influence subsequent observations. How does this formulation capture action-dependent environmental dynamics, and what justifies the joint world-action modeling interpretation in Eq. 2?
  5. Future grounding reconstructs cropped target regions while excluding frames without valid target detections from its supervision. How does this objective encourage the shared representation to capture the spatial evolution of targets and support navigation under occlusion? Could the authors provide more direct evidence of these capabilities?
Reviewer_pkAcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EndoWAM is a policy for three-DoF robotic endoscope navigation built on a LoRA-adapted Cosmos-Predict2.5 video world model. A single denoising pass yields features that feed a discrete action head and a training-only "Grounding DiT" that reconstructs target-region latents from future frames. The paper also contributes EndoMotion, 515 teleoperated phantom trajectories whose action labels are derived by a rule from Grounding DINO target tracks. On a physical endoscope across three phantom procedures, the full model clearly outperforms GR00T-N1.7 and its own variant without the grounding loss, with single-pass inference well above the rate of the iterative alternative.

Critical strengths
  1. A large, matched, hardware-validated effect of the proposed auxiliary loss. With backbone, data and budget fixed, removing the Grounding DiT drops closed-loop success from 77 to 36 of 96 trials (Table 2), and the full model also beats the variant that receives the target crop as input (Table 1).
  2. A complete system and dataset in an under-served domain. The policy is evaluated on three procedures on a physical three-DoF endoscope, runs single-pass inference several times faster than the iterative alternative (Table 4), and comes with a 515-trajectory annotated dataset.
Critical weaknesses
  1. Action labels are a deterministic rule on detector output, and that detector-plus-rule controller is never evaluated. Labels are not operator commands: target tracks are "converted into a per-frame three-DoF action label" by a zone/hysteresis rule. The policy therefore imitates a rule applied to Kalman-filtered Grounding DINO tracks, where the filter's causality is not stated, and the grounding target is a crop of the same box. The paper positions itself against visual servoing but never runs the detector, filter and rule online under the Table 1 protocol. The closest datapoint is EndoWAM-EG, which feeds the online detector's crop to the policy (Table 1), but no row executes the labelling rule itself. Until that row exists, it is unknown whether a learned world-action model is needed for this task.
  2. Neither claimed mechanism is isolated. Contributions 1-2 are a predictive dynamics prior and future grounding, yet there is no current-frame reconstruction variant, no horizon sweep and no run without the video loss (Table 2). The dynamics-only variant, without grounding, is not significantly better than GR00T-N1.7 under a Fisher exact test. It is above Qwen3AE, which shares the discrete head but not the backbone, so that contrast does not isolate the prior either. Cosmos Policy is compared on speed only (Table 4). The headline margin over GR00T-N1.7 thus arrives together with a target-supervision signal that no baseline receives.
Questions for authors

The answers to the following questions would determine whether I revisit my assessment and in which direction.

  1. Are the human operators' teleoperation commands used anywhere in training, or are all action labels the zone rule applied to Grounding DINO + Kalman tracks? What closed-loop success does that labelling pipeline achieve when run online as a controller under the Table 1 and Table 3 protocols?
  2. Please report EndoWAM with the identical Grounding DiT reconstructing the current-frame target crop, ideally at several horizons, and EndoWAM without the video loss. Does the ordering in Table 1 survive, and what horizon does the reported model use?
  3. During training, is H extracted from noised ground-truth future latents or from pure-noise future tokens, and what noise level is used at training and at test? If H sees noised ground-truth futures in training but not at test, please discuss the train/test mismatch.
  4. The robot has no roll axis. Is the test-time roll offset a physical rotation of the scope or a software rotation of the image, and if software, how are ternary bending commands mapped to the two motors at off-grid angles?
  5. Do the 32 trials per procedure share a single initial pose? Please add binomial intervals and at least a second training seed for EndoWAM and the strongest variant.
  6. How are the continuous-action baselines (Diffusion Policy, Qwen3DiT, π0.5, GR00T) trained on and executed with ternary labels, and can you report Cosmos Policy's success rate on at least one procedure?
  7. Are the evaluation phantoms the same physical units used to collect EndoMotion, and will EndoMotion, the fine-tuned detector and code be released?
Reviewer_uNfDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EndoWAM adapts a pretrained video diffusion transformer for instruction-conditioned, three-DoF endoscopic navigation. Intermediate denoising features condition a discrete action expert and an auxiliary diffusion model that reconstructs target crops from future frames; the reconstruction branch is removed at inference. EndoMotion supplies 515 teleoperated phantom trajectories across three procedures, with detector-derived action labels and rotated training views. The authors report 80.2% closed-loop success across the three phantoms, 90.1% under six environment/target shifts using a two-stage protocol, and 7.5 Hz control on an RTX 5090.

Critical strengths

The approach gives a concrete role to future target information in endoscopic policy training: target reconstruction shapes a shared video representation while adding no reconstruction branch at deployment. Extending ReconVLA's current-region supervision to predictive video features is a reasonable, task-motivated idea.

The physical evaluations make the work useful beyond architecture design. In Table 1, EndoWAM succeeds in 77/96 trials, versus 26/96 for the strongest external baseline and 59/96 for the strongest shared-backbone grounding variant. Table 2 also shows a large loss after removing grounding. These results support the effectiveness of the complete training recipe in this setting. The six controlled shifts provide additional evidence of transfer, and the reported control rate makes the design practical for further phantom studies.

Critical weaknesses

The central advance over ReconVLA-style reconstruction is supervision on future targets, yet Table 2 removes the entire grounding branch. EndoWAM-EG changes current-crop input, and the other variants change the prediction objective; none holds the reconstruction architecture fixed while changing only current versus future supervision. The temporal contribution therefore remains unresolved.

Further, Appendix B derives every action label from detected target position, visibility, and scale. No closed-loop baseline executes that same detector-and-rule controller. The strong VLA comparisons consequently do not establish how much the world-action model adds beyond its label generator. These gaps leave the main methodological advance and the need for predictive control insufficiently supported, despite the substantial phantom results.

Questions for authors

Could you compare current-frame and future-frame target reconstruction using the same Grounding DiT, backbone, loss form, and training budget? Please specify the future-frame sampling horizon; this comparison would distinguish temporal prediction from reconstruction-based grounding.

How does the Grounding DINO/Kalman controller using Appendix B's action rules perform under the same closed-loop initializations? Its success rates and failure modes would clarify the practical benefit of learning predictive control.

Comments on the paper

1 comment
Reviewer_pkAcOpenReview ↗

Thank you addressing my concerns

Thank you for the response and the additional hardware experiments, which addressed most of my concerns, I still have two points below for your further response, I will finalise my rating after your reply.

  1. Each procedure uses one reset configuration. Do you have any results from a second reset configuration? If not, how do you plan to word the generalization claim?
  2. I share Reviewer utW9's concern about "spatial evolution". The grounding targets are cropped and resized, so the loss supervises future target appearance rather than position. Do you have direct evidence that H encodes the future target position, for example a probe on H for the future box centre, compared between the full model and the current-frame variant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.