- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work propose PreMo-map, a Map of Dynamics conditioned on what the robot observes while patrolling, estimating both the rate at which pedestrians are detected at a location and the velocity distribution. The method is evaluated on real world dataset against sota baselines.
Critical strengths
-
This work address an important problem. Exisiting Maps of dynamics typically require a full range observation of the environment, although in practice a robot may only has a partial view by onboard sensors. Conditioning on current FoV is a meaningful sttep toward online deployment.
-
The proposed method uses partial observation, and achieves the promising results for the in-view setting. The ablation study helps to understand the contribution of each component. For the FoV, the observation radius is also an importance parameter and table 5 provides the evaluation results for different observatioin radius, showing the gain depends on visibility.
Critical weaknesses
-
So in current model, it does not explicitly detect positions and velocities of pedestrians in the video clip. The human states are implicitly encoded by video encoder. It could have errors accumulated in the latent representation, like missed or occluded pedestrians. It is also unclear what information in the video is essential for MoD generation. It could be the human motion or the environemnt layout for example.
-
The conditioned video clip is rendered in simulation. And the clips are renderd from the ground-truth trajectories. Will perception noise affect the performance in deployment? A real-world validation of the in-view estimate (like using the dataset of JRDB, which contains video, robot status, and human trajectories) would be helpful and strengthen the claims.
Questions for authors
Same as in the weaknesses
2 comments
Thank you for the response! Now I see how the V-JEPA is used onboard and how it capture pedestrian states without detection at inference. Thanks for adding the real-world experiment, which uses real onboard video as input rather than simulated clips. This is helpful to show the method's practicality in real world scenarios, and resolves the concerns. One more point is that, it will be more clear if the modified part get highlighted in the latest revision. I will raise my score.
We thank the reviewer for the positive assessment and for the constructive questions. We answer each point below and have updated the manuscript accordingly.
Q1. So in current model, it does not explicitly detect positions and velocities of pedestrians in the video clip. The human states are implicitly encoded by video encoder. It could have errors accumulated in the latent representation, like missed or occluded pedestrians. It is also unclear what information in the video is essential for MoD generation. It could be the human motion or the environemnt layout for example.
PreMo-map encodes the clip with a frozen V-JEPA 2.1 encoder pretrained on large-scale video, and uses no detections at inference; detections only supervise training. Whether the pedestrians in the clip and their motion reach the estimate, rather than the layout of the scene, is measured as follows:
Q2. The conditioned video clip is rendered in simulation. And the clips are renderd from the ground-truth trajectories. Will perception noise affect the performance in deployment? A real-world validation of the in-view estimate (like using the dataset of JRDB, which contains video, robot status, and human trajectories) would be helpful and strengthen the claims.
We add a real-world validation on the TBD pedestrian dataset (Wang et al., ICRA 2024; Section 4.4, Table 1), recorded by a camera on a moving robot, with pedestrian trajectories tracked by overhead cameras. Its video is real, so it carries the perception noise of an onboard camera. In view, PreMo-map outperforms all baselines on all likelihoods, with a heading likelihood 20% below the best baseline's, so the in-view gain holds on real onboard video.
We use TBD rather than JRDB because learning a map of dynamics also needs pedestrian trajectories over the whole environment, not only around the robot, and recordings at different times of day. JRDB lacks both: its trajectories cover only the robot's surroundings, and locations recorded more than once were recorded at the same time of day. TBD provides the first, but only short sessions (about 213 minutes over six days), which is why the time-of-day results rely on our grounded simulation of ATC and, new in the revision, of the Pardubice railway station (Apeltauer et al., Scientific Data 2024).
On perception in deployment: PreMo-map uses no detector at inference, so detection errors do not enter its input. The video encoder is V-JEPA 2.1, pretrained on natural video and kept frozen, so it was not fitted to rendered images.