- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper points out that EVT-Bench lets the environment wait for the policy, so inference time costs nothing, which does not match deployment. RT-EVT adds a configurable action delay and fixes the target-avatar assignment, and shows that success drops quickly with latency and that model rankings change. The authors also propose TrackRT, a small model trained with action-effect-state supervision, where the observation at request time is paired with expert trajectories from the state reached after the delay.
Critical strengths
The problem is real, and the paper gives full degradation curves over delays for three models, tied to measured device latency. This is the most reusable part of the work.
Critical weaknesses
- The benchmark does not yet show that it measures deployment latency. Success drops sharply between 100 and 110 ms and again around 200 ms for all three models despite different action conventions. The paper attributes this to the 100 ms waypoint spacing without separating it from true latency-induced degradation, does not say what the scheduler does when the delay exceeds the query period, and never defines the hold policy. The injected delay is also a fixed P50 inference time, with communication, camera and control delays excluded, so the device-specific results may not reflect real end-to-end latency. If the cliffs follow the discretization, the collapse curves and the ranking reversal are not established.
- The TrackRT claims are not controlled. TrackRT is trained with latency-aware data while the baselines are used off the shelf, and TrackRT without latency training already beats LightNav-0 at the same delay without explanation. The policy is never conditioned on the actual delay, although the right action can differ across delays, and no simple compensation baseline is tried. The request-label control in Table 3 is also far below the base model at zero latency, so it is unclear how much of the gain comes from the proposed supervision.
Questions for authors
- What does the scheduler do when the delay exceeds the query period, and how can you separate true latency-induced degradation from the artifact of the 100 ms discretization around 100–110 ms?
- Why is the policy not conditioned on the actual latency, although the optimal action may differ substantially across delays?
- Does RT-EVT represent real deployment when communication and other end-to-end delays are excluded and device-specific evaluation uses a fixed P50 latency?
- Why does TrackRT without latency training beat LightNav-0 at the same delay, and how are the baselines' waypoints converted to velocity commands?