acceptodds
Under review as a conference paper at ICLR 2027

World-Model Predictive Displays for Teleoperation

Abstract

Network latency degrades precision in teleoperation, and becomes critical in domains such as remote surgery. Predictive displays show an estimate of the present instead of the delayed camera feed, but learned image-space predictors fix the prediction horizon at training time and are valid only at that delay. We instead build a predictive display around an interactive world model, reaching the present by repeated single-step prediction from the most recent received frame. Building on DINO-WM, we predict in the feature space of a frozen encoder and decode only the displayed frame. A display thread performs one predictor and one decoder evaluation per frame regardless of the delay, while an asynchronous grounding thread re-anchors the latent context to newly arrived observations, so the compensated delay becomes a runtime quantity and jitter requires no retraining. In a within-subjects study with 22 participants on a simulated Push-T task at round-trip delays of 500–1000 ms, spanning the range where fine manipulation breaks down, compensation shortened completion time on successful trials by 23.5% and suppressed the move-and-wait strategy latency induces. Participants rated the display as more responsive but less authentic, and authenticity tracked rollout fidelity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
worldmodel, telemanipulation, teleoperation, networklatency, robotics, deeplearning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XDrGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Network latency makes teleoperation imprecise, and existing learned predictive displays only work at the single delay they were trained for. The authors instead use a DINO-WM-based world model that steps forward one prediction at a time from the latest received frame, with a separate thread re-anchoring it to new observations, so any delay or jitter is handled at runtime without retraining. In a 22-person study on a simulated Push-T task with 500–1000 ms delays, it cut completion time by 23.5% and reduced move-and-wait behavior, though users rated it less authentic than the delayed feed. The paper lists three contributions: making the compensated delay a runtime quantity rather than a training choice, an operator-in-the-loop evaluation, and support for non-constant delay by construction, the last of which the authors state is not empirically tested.

Critical strengths

The paper applies an interactive world model as a predictive display for teleoperation, addressing the long-standing problem of time delay in this setting. The authors implement the full system and evaluate it in a human-subjects experiment with 22 participants. The main strength is this attempt to put a learned world model into a teleoperation system and measure its effect on real operators.

Critical weaknesses

The paper aims to compensate for network latency in teleoperation, which it states degrades precision and fine manipulation. However, the user study reports only completion time, timeout rate, and control-input statistics; no accuracy or precision metric is evaluated. The only related quantity, end-effector path length, actually increased by about one metre under compensation, which might suggest less precise motion. This matters because the central claim of the paper, that the predictive display restores precision lost to delay, is left unsupported by the evidence presented.

DINO-WM is used with its architecture and hyperparameters essentially unchanged. The technical contribution therefore reduces to a two-thread inference scheme. The one property the paper claims distinguishes it from prior learned predictive displays, operation under variable delay without retraining, is never tested. The evaluation is confined to a simplified simulated Push-T task with fixed poses and a 2D action space, with no real sensor data or real robot, and provides little evidence for real-world settings. Taken together, the novelty of the contribution, its technical validation, and the comprehensiveness of the evaluation should be greatly improved.

Questions for authors
  1. This paper states that time delay in teleoperation degrades precision, but the reported metrics are completion time, timeout rate, and control-input statistics. If possible, it would be helpful to see a measure of placement accuracy or motion precision. It would also be helpful to know how the authors interpret the longer end-effector paths observed under compensation.

  2. In the introduction, support for variable delay is presented as a key advantage over prior learned predictive displays. However, only constant delays are evaluated. If any results under jitter or packet loss are available, even preliminary ones, they would be valuable. Otherwise, it would help to clarify on what basis this is presented as a contribution.

  3. It would be helpful to clarify whether anything in the world model itself is modified relative to DINO-WM beyond the data pipeline and the two-thread scheduling, and to state more explicitly what the authors consider the methodological contribution.

  4. Since operator performance is shown to depend on prediction fidelity, any evidence of how fidelity behaves outside the fixed-pose Push-T setting, for example on a more complex simulated scene or real camera images, would help assess the generality of the approach.

1 comment

We thank the reviewer for these concrete, specific points and address each in turn.

  1. Precision metric and end-effector path length. Agreed - we report no continuous placement-accuracy metric, only behavioral proxies. The precision claim is motivation from prior teleoperation-delay literature, not a result of this study. On the increased end-effector path length: block path length is unchanged while idle fraction falls and reversal rate rises. Read together, we interpret this as continuous closed-loop correction, not imprecision: more hand motion and more direction changes converge on an equally direct object path. We’ve added this interpretation directly to that paragraph.

  2. Variable delay. Correct that we evaluate constant delays only — the paper already flags this. We collected preliminary data: a pilot (n=5) adds per-tick jitter of +-2 and +-10 frames around the same three delays. The compensation advantage holds under jitter (HR = 1.69, p = .005) with no evidence it depends on jitter amplitude. Full design, tables, and caveats appropriate to n=5 are in a new appendix section; we present this as preliminary feasibility evidence, not a powered test, and the contribution claim now points there instead of standing untested.

  3. What’s modified relative to DINO-WM, and the methodological contribution. To be precise: the world-model architecture itself (encoder, predictor, decoder) is unmodified DINO-WM. Our contribution is the two-thread display/grounding scheduling procedure that decouples per-frame inference cost from the delay, the feasibility/horizon-bound analysis characterizing exactly when this is possible given measured component latencies and the resulting property that delay is a runtime rather than a training-time quantity.

  4. Fidelity beyond fixed-pose Push-T. We have no evidence to offer here — this is a genuine limitation, already stated. We don’t have grounds to predict how fidelity would behave on a more complex scene or real camera images and would rather say so than speculate.

Reviewer_U57FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts DINO-WM into a predictive display for teleoperation with delayed video and immediate action execution. A display thread predicts one latent step and decodes one image per frame, while an asynchronous grounding thread rolls newly received observations forward and replaces the shared context. The same model serves different delays through its rollout length. A counterbalanced within-subjects study with 22 participants evaluates three constant delays on a restricted simulated Push-T task. The authors report faster successful trials and less idle input, alongside lower perceived display authenticity and worsening prediction fidelity at longer delays.

Critical strengths

The useful contribution is the connection between latent world modeling and an operator-facing display. Separating grounding from image decoding makes interactive prediction plausible without a decoder call at every intermediate rollout step. The prediction model and separately trained decoder come from DINO-WM; the contribution here lies in their use within an asynchronous feedback interface and its evaluation.

The operator study offers useful evidence of both benefit and cost. Twenty of 22 participants completed successful trials faster with compensation; the authors report a 17.7% reduction in wall-clock time on successful trials, alongside greater responsiveness and lower display authenticity. Timeout rates did not reliably improve, and fidelity-performance correlations do not establish causality. Even with those limits, the working interface and the observed interaction tradeoff provide sufficient value for acceptance as a bounded feasibility study. The timing bound needs correction and the effect of task pacing remains unresolved, but neither issue erases this contribution.

Critical weaknesses

The stated horizon bound is not supported by Algorithm 1. In an idealized schedule, with k = 8 and predictor latency one fifth of a frame period, Eq. 4 gives G = 2. A pass anchored at frame -8 finishes at frame 2; at frame 3 the display still uses that anchor, yielding 11 steps, exceeding k + G = 10. Encoding and scheduling costs further complicate the claimed bound.

Appendix B.8 reports about 20 Hz for baseline versus 19.15-17.89 Hz under compensation. Dividing steps by 20 does not remove the changed human-system timing. The reported 17.7% wall-clock improvement remains promising; a matched-rate comparison would help separate prediction from altered task pacing.

Questions for authors

Can you provide a corrected worst-case horizon bound and measured grounding intervals and anchor ages, including encoding and shared-GPU scheduling?

Does an analysis of wall-clock completion times, retaining censored trials, support the same conclusion? Can you separate the effect of prediction from different task pacing using a baseline matched to the compensated rate? These requests concern the existing system claims; extending the study to realistic robots is future work.

Reviewer_pxMeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a world-model-based predictive display to compensate for visual feedback latency in robot teleoperation. The authors build on DINO-WM and introduce a two-thread architecture for frame prediction and asynchronous grounding. They evaluate the system on a simulated Push-T task .

Critical strengths

The authors conduct a user study in an idealized simulation setting and demonstrate some improvements in task completion time and perceived responsiveness.

Critical weaknesses

This paper reads more like a course project than a research paper suitable for ICLR. The authors simply adopt an existing model and train and evaluate it in a highly unrealistic setting with little practical relevance, making the overall contribution almost meaningless. The research problem itself is also poorly motivated and lacks a convincing rationale. Furthermore, the experimental results are extremely limited.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.