acceptodds
Under review as a conference paper at ICLR 2027

Learning Spatio-Temporal Continuous Presence and Motion Fields from Onboard Video

Abstract

Mobile robots operating in crowded environments depend on an accurate model of human motion for efficient and safe planning and navigation. Maps of dynamics (MoDs) model motion as a persistent property of the environment, representing either its direction or its presence. We introduce PreMo-map, an MoD conditioned on what the robot observes while patrolling, estimating both the rate at which pedestrians are detected at a location and the distribution of their velocities there. Both quantities are evaluated at continuous coordinates and learned from individual detections rather than from a precomputed grid, and positions inside and beyond the field of view are estimated by separate heads. Measured on a large-scale real-world dataset, PreMo-map outperforms state-of-the-art MoDs within the field of view and remains competitive over the rest of the environment, while also returning a rate that published MoDs do not provide.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Maps of Dynamics, Spatio-temporal Dynamics, Human Motion Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CfM1OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work propose PreMo-map, a Map of Dynamics conditioned on what the robot observes while patrolling, estimating both the rate at which pedestrians are detected at a location and the velocity distribution. The method is evaluated on real world dataset against sota baselines.

Critical strengths
  1. This work address an important problem. Exisiting Maps of dynamics typically require a full range observation of the environment, although in practice a robot may only has a partial view by onboard sensors. Conditioning on current FoV is a meaningful sttep toward online deployment.

  2. The proposed method uses partial observation, and achieves the promising results for the in-view setting. The ablation study helps to understand the contribution of each component. For the FoV, the observation radius is also an importance parameter and table 5 provides the evaluation results for different observatioin radius, showing the gain depends on visibility.

Critical weaknesses
  1. So in current model, it does not explicitly detect positions and velocities of pedestrians in the video clip. The human states are implicitly encoded by video encoder. It could have errors accumulated in the latent representation, like missed or occluded pedestrians. It is also unclear what information in the video is essential for MoD generation. It could be the human motion or the environemnt layout for example.

  2. The conditioned video clip is rendered in simulation. And the clips are renderd from the ground-truth trajectories. Will perception noise affect the performance in deployment? A real-world validation of the in-view estimate (like using the dataset of JRDB, which contains video, robot status, and human trajectories) would be helpful and strengthen the claims.

Questions for authors

Same as in the weaknesses

2 comments

We thank the reviewer for the positive assessment and for the constructive questions. We answer each point below and have updated the manuscript accordingly.

Q1. So in current model, it does not explicitly detect positions and velocities of pedestrians in the video clip. The human states are implicitly encoded by video encoder. It could have errors accumulated in the latent representation, like missed or occluded pedestrians. It is also unclear what information in the video is essential for MoD generation. It could be the human motion or the environemnt layout for example.

PreMo-map encodes the clip with a frozen V-JEPA 2.1 encoder pretrained on large-scale video, and uses no detections at inference; detections only supervise training. Whether the pedestrians in the clip and their motion reach the estimate, rather than the layout of the scene, is measured as follows:

  • Removing the clip (Table 3). In view, the heading error rises from 43.6° to 82.8°, against 90° for chance, and the heading likelihood by 0.366 nats. The in-view heading is therefore recovered from the clip.
  • Human motion or layout (Table 4). Replacing the camera clip with one fixed clip of the same environment, an empty corridor or a crowd, brings the in-view heading error to 89.5° and 89.2°. A fixed clip shows the same layout but not the pedestrians present at the query, so the estimate is read from the current pedestrians and not from the layout.
  • Content of the frozen features (new Appendix A5, Table A2). On our clips, probes on the frozen features recover whether a pedestrian is visible (AUC 0.980), how many are visible (R2R^2 0.822) and their direction of motion (35.0°35.0\degree error). The same probes on the robot's pose reach 0.802, 0.075 and 97.7°97.7\degree.

Q2. The conditioned video clip is rendered in simulation. And the clips are renderd from the ground-truth trajectories. Will perception noise affect the performance in deployment? A real-world validation of the in-view estimate (like using the dataset of JRDB, which contains video, robot status, and human trajectories) would be helpful and strengthen the claims.

We add a real-world validation on the TBD pedestrian dataset (Wang et al., ICRA 2024; Section 4.4, Table 1), recorded by a camera on a moving robot, with pedestrian trajectories tracked by overhead cameras. Its video is real, so it carries the perception noise of an onboard camera. In view, PreMo-map outperforms all baselines on all likelihoods, with a heading likelihood 20% below the best baseline's, so the in-view gain holds on real onboard video.

We use TBD rather than JRDB because learning a map of dynamics also needs pedestrian trajectories over the whole environment, not only around the robot, and recordings at different times of day. JRDB lacks both: its trajectories cover only the robot's surroundings, and locations recorded more than once were recorded at the same time of day. TBD provides the first, but only short sessions (about 213 minutes over six days), which is why the time-of-day results rely on our grounded simulation of ATC and, new in the revision, of the Pardubice railway station (Apeltauer et al., Scientific Data 2024).

On perception in deployment: PreMo-map uses no detector at inference, so detection errors do not enter its input. The video encoder is V-JEPA 2.1, pretrained on natural video and kept frozen, so it was not fitted to rendered images.

Thank you for the response! Now I see how the V-JEPA is used onboard and how it capture pedestrian states without detection at inference. Thanks for adding the real-world experiment, which uses real onboard video as input rather than simulated clips. This is helpful to show the method's practicality in real world scenarios, and resolves the concerns. One more point is that, it will be more clear if the modified part get highlighted in the latest revision. I will raise my score.

Reviewer_k7LyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PreMo-map predicts pedestrian detection rates and velocity distributions at arbitrary locations, conditioned on a short onboard video, robot pose, and time. It jointly learns these quantities through a marked point-process likelihood. The data come from replaying the ATC tracks in Gazebo and rendering egocentric clips from a simulated patrolling robot. Inside the FoV, PreMo-map beats NeMo-map and the grid-based MoDs on flow and beats FreMEn on presence. Beyond the FoV it is worse than NeMo-map on flow and on par with FreMEn on presence.

Critical strengths

*A coherent probabilistic formulation. Jointly modeling rate and velocity distinguishes “how likely someone is to be here” from “how someone here moves.”

  • Continuous spatial queries avoid committing predictions to a fixed map grid.

  • The ablations are informative.. It demonstrates actual video use. Removing or replacing the clip severely harms in-view heading estimation, supporting the central conditioning claim.

Critical weaknesses
  • Simulation realism. The video is Gazebo-rendered pedestrians replayed from 1 Hz tracks, so the clip effectively encodes clean, synthetic agent positions. There is no real onboard video, no detector noise, and no sim-to-real analysis. It is unclear whether the gain survives real imagery, occlusion, and imperfect detections.

  • Benefits are largely confined to visible locations. Beyond view, NeMo-map has better joint flow likelihood (1.848 versus 1.959). Since 97.7% of evaluated detections are outside view, the model does not improve overall flow likelihood; the appendix confirms this on every test day.

  • A crucial baseline is missing. Compare against a pedestrian detector/tracker using current observations inside view, combined with a simple constant volecity model outside view. That would test whether the learned field improves over a straightforward hybrid

Questions for authors
  1. How does PreMo-map compare in view against a detection + tracking + constant-velocity (or learned forecaster) baseline, and against Online CLiFF-map updated with the in-view detections?
  2. Can you evaluate on real onboard video, to estimate the sim-to-real gap?
  3. Why is the beyond-view branch worse than NeMo-map despite having strictly more inputs? Can the clip be made useful beyond the FoV?
1 comment

We thank the reviewer for the careful reading. We answer each question below and have updated the manuscript accordingly.

Q1. How does PreMo-map compare in view against a detection + tracking + constant-velocity (or learned forecaster) baseline, and against Online CLiFF-map updated with the in-view detections?

We add both baselines suggested.

  • Detection + tracking + constant velocity (new Appendix A6, Table A3). It reads the same monocular frames as PreMo-map, detects and tracks the pedestrians in them, and predicts each one's velocity as a distribution, evaluated with the same likelihoods on the same in-view detections. It is therefore directly comparable in both input and metrics.

    However, while PreMo-map returns a velocity distribution at every position in view, the baseline only reports pedestrians it holds a consistent track on, missing about a third of the in-view detections (pedestrians who just entered the view, were missed by the detector or whose track was lost). A learned trajectory forecaster would replace the constant-velocity model but still take these tracks as input, so it would miss the same detections.

    Over the three ATC test days, PreMo-map achieves the lower joint and heading likelihoods on every day: 1.450 and 1.098 nats pooled, against 1.610 and 1.301 for the tracker. On speed, the tracker is ahead on one of the three days and over the pooled days (0.297 against 0.328), since it measures speed on the pedestrian it follows; PreMo-map is ahead on the other two. Beyond view, where 97.7% of the detections lie, a constant-velocity model can only extrapolate pedestrians it has seen and answers almost nowhere, whereas a map of dynamics answers at every position.

  • Online CLiFF-map updated with in-view detections (new rows of Table 1, Section 4.3, Table A1). We update the map during each evaluation day with the in-view detections. In view, PreMo-map remains ahead on the joint and heading likelihoods at both sites: 1.450 and 1.098 nats against 1.806 and 1.468 at ATC, and 1.389 and 0.723 against 3.890 and 2.365 at Pardubice (the second site we added in revision (Apeltauer et al., Scientific Data 2024)). At ATC, the updated map improves on speed, 0.302 against 0.329, and also beyond view, where it receives no observation. Since the update lets the map learn from the days it is evaluated on, its gains reflect adaptation to the evaluation day rather than use of the current observation: the update improves the whole-environment likelihood on one Sunday (2.425 to 2.024) and degrades it on the Wednesday that matches the training day (1.730 to 2.062; Table A1). At Pardubice, the update degrades the map on every channel and every day, to a joint likelihood above that of the uniform flow (3.890 against 2.560 in view).

Q2. Can you evaluate on real onboard video, to estimate the sim-to-real gap?

We add a real-world validation on the TBD pedestrian dataset (Wang et al., ICRA 2024; Section 4.4, Table 1), recorded by a camera on a moving robot, with pedestrian trajectories tracked by overhead cameras. In view, PreMo-map outperforms all baselines on all likelihoods, with a heading likelihood 20% below the best baseline's, within the range of the simulated sites (16% at ATC, 39% at Pardubice), so the in-view gain holds on real onboard video.

Q3. Why is the beyond-view branch worse than NeMo-map despite having strictly more inputs? Can the clip be made useful beyond the FoV?

The gap comes from the position and time representation of our readout, not from the clip. Within our architecture, the model conditioned on position and time alone reaches a beyond-view heading likelihood of 1.674 nats on the first test day, against 1.586 for NeMo-map (Tables 3 and 5), and removing the clip from the full model changes the beyond-view heading likelihood by only 0.018 nats. NeMo-map encodes position with a learned spatial feature grid and time with a periodic encoder, whereas our readout embeds position with an MLP. We note that beyond view PreMo-map is second only to NeMo-map on the joint flow likelihood at ATC and Pardubice, ahead of CLiFF-map and Online CLiFF-map (Table 1), and matches FreMEn on presence (Table 2).

The spatial reach analysis (Table 5) helps address this question. The clip's contribution follows visibility rather than proximity: inside the field of view, PreMo-map improves on NeMo-map's heading likelihood at every radius, by 0.205 nats even at the 8 m range of the camera, whereas just outside it, within 2 m of the robot, it is behind by 0.102. A clip informs the positions it shows.

Reviewer_H1TZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PreMo-map, an observation-conditioned map of dynamics that jointly estimates pedestrian detection intensity and conditional velocity distributions at continuous spatial coordinates. A frozen video encoder processes a short egocentric clip, which is fused with robot pose and time. Separate readouts estimate rate and flow inside and outside the camera’s field of view. The model is trained from individual detections using a marked Poisson-process likelihood and a two-stage curriculum.

Critical strengths
  1. PreMo-map jointly predicts pedestrian presence rate and velocity flow from egocentric video, pose, and time in a continuous spatial field.
  2. It outperforms existing methods inside the camera’s field of view, with ablations confirming the value of onboard video.
Critical weaknesses
  1. The incremental methodological contribution is not sufficiently isolated from the benefit of additional observations. Continuous neural flow fields and semi-wrapped velocity mixtures are established by NeMo-map; the main additions here are video conditioning and a rate head. However, the principal baselines do not receive the same current visual evidence.
  2. The joint likelihood decomposes additively into rate and flow terms, yet no matched comparison against separately trained predictors establishes the benefit of joint estimation.
  3. The demonstrated benefit is narrow relative to the intended whole-environment robotics setting. All visual observations are rendered in one reconstructed environment, using a prescribed robot patrol.
  4. There is no validation on real onboard imagery or a held-out environment. Moreover, 97.7% of evaluated detections lie outside the field of view, where PreMo-map has worse flow NLL than NeMo-map; Appendix A1 reports worse whole-environment flow NLL on every test day. The in-view improvement therefore does not establish a net improvement in the full map.
Questions for authors
  1. Is the gain from video or extra observations? Add matched baselines.
  2. Does joint training beat separate rate and flow models?
  3. Does the method generalize to real video, new environments, and patrol routes?
  4. What is the whole-map performance when 97.7% of detections are out of view?
3 comments

We thank the reviewer for the detailed review. We answer each question below and have updated the manuscript accordingly.

Q1. Is the gain from video or from extra observations? The principal baselines do not receive the same current visual evidence; add matched baselines.

Existing MoDs are fitted from accumulated trajectories and queried by position and time. None of them accepts a current observation, and no matched version that reads video exists. Conditioning a MoD on the robot's current view is the gap this work addresses.

To separate the contribution of video from that of extra observations, we add a baseline that receives the robot's current observations, as suggested by Reviewer k7Ly: Online CLiFF-map updated with in-view detections (new rows of Table 1, Section 4.3, Table A1). It receives more than PreMo-map: the current observations and a memory of everything observed that day. In view, PreMo-map remains ahead on the joint and heading likelihoods at both sites: 1.450 and 1.098 nats against 1.806 and 1.468 at ATC, and 1.389 and 0.723 against 3.890 and 2.365 at Pardubice (the railway station we add as a second simulated site). At ATC, the updated map improves on speed; at Pardubice, the update degrades it on every channel, to a joint likelihood above that of the uniform flow (3.890 against 2.560 in view). We attribute both effects to its adaptation to the evaluation day rather than to the current observation, since it is fitted in part on the day it is evaluated on.

We also add a tracker as a baseline (as suggested by Reviewer k7Ly) that receives exactly the same visual evidence, and extracts the current motion explicitly: it detects and tracks the pedestrians in view and predicts each one's velocity with a constant-velocity model, evaluated with the same likelihoods on the same detections (new Appendix A6, Table A3). However, while PreMo-map returns a velocity distribution at every position in view, the baseline only reports pedestrians it holds a consistent track on, missing about a third of the in-view detections, such as people entering the field of view after the clip ends. Over the three ATC test days, PreMo-map achieves the lower joint and heading likelihoods, 1.450 and 1.098 nats pooled against 1.610 and 1.301. The tracker is ahead on speed over the pooled days (0.297 against 0.328), which it measures on the pedestrian it follows.

About the gain coming from the video or the extra information, our ablations isolate the clip within the same architecture and training data (Tables 3 and 4): removing it raises the in-view heading error from 43.6° to 82.8° (90° is chance), and fixing it at inference brings the error to chance, so the in-view gain comes from the current clip.

Q2. The likelihood decomposes additively; does joint training beat separate rate and flow models?

Our claim is that one network estimates both quantities, not that joint training makes either one more accurate. Rate and flow are the two factors of one marked point process, Λ = λ·p (Eq. 1). Their likelihoods therefore add (Eq. 7), and one network can be fitted to both under a single objective. Rate and flow share the frozen encoder and the fusion stage, which is trained under this objective, and are read out by separate branches, as described in Sections 3.2 and 3.3.

Q3. Does the method generalize to real video, new environments and patrol routes?

New environments. We add a second site, the Pardubice railway station (Apeltauer et al., Scientific Data 2024), built with the same grounded simulation (Section 4.1, Table 1, Section 4.3, Table A1). We evaluate it in two ways.

  • Trained and evaluated at Pardubice, with every baseline fitted on its training day, the pattern of ATC holds and the in-view gain is larger: PreMo-map achieves the lowest joint and heading likelihoods, 1.389 and 0.723 nats against 1.786 and 1.183 for NeMo-map, a heading likelihood 39% lower, against 16% at ATC. Beyond view, NeMo-map leads, and PreMo-map is second on the joint likelihood.
  • Trained on ATC and evaluated at Pardubice without retraining (Table 1, Section 4.3). Maps of dynamics fitted on a site's trajectories are functions of its coordinates and cannot be carried to another site, whereas PreMo-map also reads the clip. For this setting we train it at ATC without the position and time-of-day inputs, which are specific to each site, and evaluate it in view only, since beyond view the estimate rests on position and time alone, which do not carry over between sites. In view, its heading likelihood (0.942 nats) is lower than that of every baseline fitted on Pardubice's own training day, NeMo-map included (1.183). The speed does not transfer, so its joint likelihood (1.921) is above NeMo-map's (1.786), while below CLiFF-map's (2.060). The heading read from the clip therefore transfers to an unseen environment.

Real video. We add a real-world validation on the TBD pedestrian dataset (Wang et al., ICRA 2024; Section 4.4, Table 1), recorded by a camera on a moving robot, with pedestrian trajectories tracked by overhead cameras. TBD provides pedestrian trajectories over the whole environment, but only short sessions (about 213 minutes over six days). In view, PreMo-map outperforms all baselines on all likelihoods, with a heading likelihood 20% below the best baseline's, so the in-view gain holds on real onboard video. Its robot also moves along its own routes through the hall rather than our simulated patrol loop, so the result holds beyond the patrol route used in training and evaluation at ATC and Pardubice.

Q4. What is the whole-map performance when 97.7% of detections are out of view?

Since 97.7% of the detections lie beyond the field of view at ATC, and 97.8% at Pardubice, the whole-environment result follows the beyond-view column of Table 1; the per-day figures for every site are reported in Table A1. At ATC and Pardubice, NeMo-map is ahead of PreMo-map on every day, and PreMo-map is ahead of CLiFF-map and Online CLiFF-map. We note that beyond view, PreMo-map is second only to NeMo-map on the joint likelihood at both sites, while being the only method that also estimates the rate. Within the field of view, its improvement over NeMo-map exceeds its deficit beyond it at both sites: on heading, 16% lower in view against 8% higher beyond at ATC, and 39% lower against 20% higher at Pardubice. At TBD, recorded with real onboard video, PreMo-map is ahead of every baseline beyond view on the joint likelihood, NeMo-map included, and has the lowest whole-environment likelihood on every day (Tables 1 and A1).

Reviewer_H1TZOpenReview ↗

Assessment Following the Author Rebuttal

Thank you for the detailed response. Nevertheless, I still believe that the paper primarily introduces video conditioning into an existing continuous dynamic map modeling framework, with relatively limited methodological novelty. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.