acceptodds
Withdrawn from ICLR 2027

On the move: Latency-aware Embodied Visual Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied visual tracking (EVT) is inherently latency-sensitive. While a robot computes its next action, the target keeps moving, so observations grow stale before actions take effect. Existing benchmarks, however, advance the environment only after the policy returns its action, charging nothing for decision time. We introduce RT-EVT, a latency-aware tracking benchmark that evaluates policies under configurable action latency. Accounting for deployment latency substantially changes tracking performance. For example, LightNav-0's tracking success rate drops from a SOTA 89.8% to 22.9% under 200 ms latency, even worse than a 5× smaller model. We propose TrackRT, a compact 0.9B vision-language navigation model designed for real-time tracking. We further introduce latency-aware training, implemented through action-effect-state supervision. The policy is trained against expert trajectories from the states in which its actions take effect. On the Jetson Orin NX, TrackRT achieves a mean inference latency of 145.0 ms versus 1201.0 ms for LightNav-0, with corresponding RT-EVT STT success rates of 65.3% and 0.5% respectively. Controlled ablations across different latency settings further confirm the benefit of latency-aware training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied AI, robot learning, vision-language models, benchmark and evaluation, inference latency, vision-language navigation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nqiSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that EVT-Bench lets the environment wait for the policy, so inference time costs nothing, which does not match deployment. RT-EVT adds a configurable action delay and fixes the target-avatar assignment, and shows that success drops quickly with latency and that model rankings change. The authors also propose TrackRT, a small model trained with action-effect-state supervision, where the observation at request time is paired with expert trajectories from the state reached after the delay.

Critical strengths

The problem is real, and the paper gives full degradation curves over delays for three models, tied to measured device latency. This is the most reusable part of the work.

Critical weaknesses
  • The benchmark does not yet show that it measures deployment latency. Success drops sharply between 100 and 110 ms and again around 200 ms for all three models despite different action conventions. The paper attributes this to the 100 ms waypoint spacing without separating it from true latency-induced degradation, does not say what the scheduler does when the delay exceeds the query period, and never defines the hold policy. The injected delay is also a fixed P50 inference time, with communication, camera and control delays excluded, so the device-specific results may not reflect real end-to-end latency. If the cliffs follow the discretization, the collapse curves and the ranking reversal are not established.
  • The TrackRT claims are not controlled. TrackRT is trained with latency-aware data while the baselines are used off the shelf, and TrackRT without latency training already beats LightNav-0 at the same delay without explanation. The policy is never conditioned on the actual delay, although the right action can differ across delays, and no simple compensation baseline is tried. The request-label control in Table 3 is also far below the base model at zero latency, so it is unclear how much of the gain comes from the proposed supervision.
Questions for authors
  • What does the scheduler do when the delay exceeds the query period, and how can you separate true latency-induced degradation from the artifact of the 100 ms discretization around 100–110 ms?
  • Why is the policy not conditioned on the actual latency, although the optimal action may differ substantially across delays?
  • Does RT-EVT represent real deployment when communication and other end-to-end delays are excluded and device-specific evaluation uses a fixed P50 latency?
  • Why does TrackRT without latency training beat LightNav-0 at the same delay, and how are the baselines' waypoints converted to velocity commands?
Reviewer_BzjTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how inference latency affects embodied visual tracking when observations become stale before actions take effect. It introduces a benchmark that incorporates action delays and a compact tracking policy trained using supervision aligned with the states in which actions are executed. This connects the evaluation protocol with a training strategy for anticipating motion during computation. Experiments across delay settings, hardware measurements, controlled training comparisons, and robot trials demonstrate the impact of latency and the benefits of the proposed approach.

Critical strengths
  • The benchmark addresses a consequential evaluation problem and audits identity consistency across all 4,215 episodes (Appendix A.3). At a common 200 ms delay, TrackRT substantially exceeds the strongest baseline across all three tasks (Table 16), separating robustness from inference speed.
  • Table 3 and Appendix B.6 isolate label timing using identical delayed rollouts, inputs, sample counts, and update budgets. Action-effect supervision improves success over base training by 15.1/10.0/12.4 percentage points at 200 ms, providing concrete evidence for the training contribution.
Critical weaknesses
  • The simulator's physical interpretation remains insufficiently established. Figure 4 assigns a 25 ms base-integration budget to a 100 ms logical macro-step and leaves physics-substep durations runtime-defined. Together with the sharp 100–110 ms performance cliff (Section 5.2), this makes it difficult to distinguish deployment-relevant delay sensitivity from controller/scheduler effects.
  • Hardware-derived comparisons inject a fixed P50 inference delay, excluding camera age and robot control (Appendix C.2). The 30 trials per backend in one corridor support a latency benefit for TrackRT, but do not validate the comparative ranking of models on a physical robot (Appendix C.3).
Questions for authors
  1. Clarification: How do robot integration, person motion, and physics substeps map to the same physical time in Figure 4? Does the ranking reversal persist when waypoint/control discretization changes?
  2. Clarification: For delays exceeding the 100 ms query interval, how are inference throughput, pending requests, and command replacement handled? Are the injected device-delay conditions realizable by the measured single-device execution schedule?
Reviewer_iVrSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses latency in embodied visual tracking by introducing a latency-aware benchmark (RT-EVT), a compact 0.9B model (TrackRT), and action-effect-state supervision for training on states where actions actually take effect. However, the core idea of accounting for action latency and training on latency-shifted states appears incremental rather than fundamentally novel. The work’s contribution seems largely to combine established ideas—real-time inference optimization, latency-aware evaluation, and state-based supervision—within a specific tracking benchmark, with limited evidence of broader methodological novelty.

Critical strengths

The proposed latency-aware benchmark provides a more realistic evaluation setting, and the reported experiments highlight a substantial gap between conventional and latency-aware performance assessment.

Critical weaknesses
  1. The paper addresses a well-studied problem: the impact of action latency on embodied control and the need to account for delayed action effects during training and evaluation. The core idea is largely an application of established principles to visual tracking, and the paper does not sufficiently differentiate its contributions from prior work. The experiments also fail to convincingly establish a substantial methodological advance beyond this existing line of research.

  2. The reported gains are heavily tied to inference speed and the proposed latency-aware benchmark, making it difficult to disentangle the benefits of a smaller, faster model from those of the proposed training method. More rigorous comparisons with latency-matched baselines and stronger evidence across diverse tracking scenarios are needed to demonstrate that the approach offers generalizable improvements beyond this particular setup.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.