acceptodds
Under review as a conference paper at ICLR 2027

StreamDrive: Streaming Perception for End-to-End Autonomous Driving via Delay-Conditioned Feature Extrapolation

Abstract

End-to-end autonomous driving systems achieve strong offline benchmarks but ignore a real-world constraint: by the time inference completes, the world has moved on and predictions are stale. We present StreamDrive, a streaming perception framework that equips end-to-end driving models with delay-aware feature extrapolation. StreamDrive corrects latent instance features rather than individual task outputs: a single delay-conditioned residual propagates through the entire task stack: detection, motion forecasting, and planning. A timestamp-aware temporal queue replaces frame-indexed storage with wall-clock timestamps, keeping instances matched under variable frame rates and irregular sensor timing. StreamDrive decomposes delay compensation into a constant-velocity baseline that captures first-order motion, plus a FiLM residual that learns non-linear, delay-dependent corrections conditioned on scene context and ego kinematics. On nuScenes, StreamDrive improves streaming detection mAP by  pp at real-time () playback over the SparseDrive-S baseline. Streaming planning collision rate drops by 9.4% relative at and 14.2% at playback speed, with consistent gains across all tasks and playback speeds. We also introduce a multi-task streaming evaluation framework for end-to-end driving, including streaming metrics for motion forecasting and the Plan Consistency Score (PCS), a time-adjacent measure of plan stability under latency that complements accuracy metrics. StreamDrive adds less than 1% parameter overhead and less than 1ms of inference latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous driving, Computer vision, Delay compensation, Streaming Perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uXnxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamDrive, a streaming perception framework that compensates for inference delay by extrapolating latent instance features in end-to-end autonomous driving. It combines a constant-velocity prior with a learned residual conditioned on scene context and ego state. The paper also introduces a multi-task streaming evaluation protocol and the Plan Consistency Score (PCS). Experiments show that StreamDrive improves perception, motion forecasting, and planning performance under streaming inference.

Critical strengths
  1. The proposed streaming feature compensation is simple and effective, addressing inference delay across perception, motion forecasting, and planning within a unified framework.

  2. The method achieves consistent improvements across multiple tasks while introducing limited additional parameters and inference overhead.

  3. The experimental evaluation is reasonably comprehensive, including ablation studies, comparisons with alternative methods, and tests under different latency conditions. The results support the effectiveness and practical applicability of the proposed approach.

Critical weaknesses
  1. It is unclear how the training targets are constructed. The model is trained with delays sampled from [0, 0.5] s, but nuScenes provides standard 3D box annotations at 2 Hz. For a delay of 200 or 400 ms, where does the corresponding future ground truth come from? The paper says that labels are not interpolated. Transforming the existing boxes into a future ego frame would account for ego motion, but would not update the positions of moving objects. The same question applies to the fine-grained delays used in evaluation. I could not work out the actual input–target pairing from the current description.

  2. The zero-delay behavior needs clarification. Zero initialization makes the residual zero at the start of training, but it does not ensure that it stays zero at dt=0 after training. Is there a bypass or an explicit constraint for this case? The matching offline scores are reassuring, but they do not explain how the claimed exact preservation of the original outputs is achieved.

  3. The experiments are comprehensive, but the discussion mostly repeats the numbers in the tables. A deeper analysis of the observed trends would help explain the method's behavior. Also, qualitative examples are missing. A few detection visualizations before and after compensation would help show what the module corrects and where it still struggles.

Questions for authors

Please address the questions raised in Weaknesses.

1 comment

We thank the reviewer for the constructive assessment; both technical concerns are valid. After receiving the reviews, we audited the evaluation code against the manuscript and removed every claim unsupported by the protocol. The response contains corrections, mechanisms that can be verified in our implementation, and existing measurements.

Corrected protocol (context for all answers). We report three playback speeds. Replaying the nuScenes keyframe timeline at 1×/2×/4× gives playback intervals of 0.5/0.25/0.125 s. Playback does not interpolate labels; every scored label is an annotated keyframe. Each model runs at measured latency, and its output is paired with the first annotated keyframe after it is ready. The revision removes the "≈0.1/0.2/0.5 s effective delay" mapping, the Δ = t_small + s·dt model, Fig. 1(a), Table 9, Sec. 5.7, and the appendix delay-generalization study. Keyframe-only scoring cannot realize sub-0.5 s offsets; those results did not measure the claimed delays.

W1 (training-target and fine-grained-delay construction). Your interpretation is largely consistent with the implementation; the revision now describes it precisely. During training, each sample is paired with a real future keyframe, 0.5 s later. Agent boxes and trajectories come from that keyframe's annotations, so moving agents are represented by real future labels rather than by transforming current boxes. The future ego frame comes from recorded odometry. Each sample's delay is the actual elapsed time to its paired keyframe. It is discrete and determined by the keyframe spacing, not sampled continuously as "dt ~ U[0, 0.5] s"; we have corrected that statement in the paper. Evaluation follows the same no-synthesis rule: a prediction from keyframe i is scored against the first annotated keyframe strictly after it is ready, accounting for the model's measured latency. Realized offsets are therefore quantized to 0.5 s increments. The reviewer is right that ego-motion transforms alone would not move agents; this is why agent states come from real future annotations.

W2 (zero-delay behavior after training). Agreed: zero-initialization guarantees a zero residual only at the start of training, so "preserved exactly by construction" was too strong. The offline rows are identical because the dt = 0 path bypasses the correction: constant-velocity extrapolation is an exact no-op at dt = 0, and the FiLM branch is not applied. The revision will state this explicitly and also gate the residual with g(dt) = 1 − exp(−dt/τ). Since g(0) = 0 exactly, the architecture guarantees identity at dt = 0 for any training trajectory and any random seed; it does not depend on initialization or on the bypass convention.

W3 (deeper analysis; qualitative examples). These are fair requests. The revision will add: (i) a static-versus-moving split for streaming detection and motion, computed from existing per-scene arrays, to show where the correction has the greatest effect; (ii) an analysis of accelerating and turning agents, where the constant-velocity prior is less accurate and the learned residual should matter most, including failure cases; (iii) before-and-after examples under latency for detection, forecasting, and planning, showing failures such as occluded or abruptly maneuvering agents as well as successes; and (iv) discussion that interprets the results rather than restating the tables. For example, we will discuss why detection gains plateau in absolute mAP while planning gains grow with playback speed (visibility limits versus accumulating ego displacement), and why motion gains are smaller (the forecast head already models some temporal information). Two existing analyses are also relevant: paired per-scene bootstrap intervals over 150 validation scenes, and the irregular-timing study, where StreamDrive's advantage grows when dropped frames violate frame-indexed assumptions.

What remains valid. All methods were evaluated using the same pairing rules. Existing measurements therefore support the within-protocol conclusions: improvements over SparseDrive-S on all six streaming accuracy metrics at every playback speed, with paired-bootstrap intervals excluding zero; transfer to a second backbone (UniAD); and graceful degradation under jitter, dropped frames, and nonuniform timing. These results use 0.55 M parameters and add less than 1 ms of overhead. The revision will release per-frame predictions, per-model latency logs, and configurations so each reported condition can be reproduced.

We believe the corrected target/protocol description, the architectural zero-delay identity, and the added analyses address your three weaknesses directly; thank you for the careful and collegial review.

Reviewer_v8orOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StreamDrive, a latency-aware framework for end-to-end autonomous driving. By combining timestamp-aware temporal queues, constant-velocity extrapolation, and FiLM-based feature correction, the method compensates for inference delays across detection, motion forecasting, and planning. Experiments on nuScenes demonstrate consistent improvements with minimal computational overhead.

Critical strengths
  1. The paper addresses an important practical limitation of existing end-to-end driving systems: inference-induced prediction staleness.
  2. The unified feature-level correction is lightweight and improves multiple downstream tasks simultaneously.
Critical weaknesses
  1. The paper reports results at fine-grained delays (0.05–0.5 s) while claiming to use only real nuScenes keyframe annotations without interpolation. Given the sparse annotation frequency, how ground truth is obtained at these intermediate timestamps remains unclear. Moreover, the reported inference latency (156.8 ms) appears inconsistent with the claimed effective delays of approximately 0.1 s at 1× and 0.5 s at 4× under the stated latency formulation.

  2. The main components, including constant-velocity extrapolation, timestamp-aware temporal modeling, and FiLM conditioning, are established techniques. The novelty primarily lies in their integration into a shared feature representation. Stronger comparisons with existing latency-aware methods (e.g., LASP) and learned output-level correction baselines would better substantiate the claimed advantages.

  3. The paper attributes exact offline performance preservation to zero initialization. However, zero initialization only guarantees an identity mapping before training, not necessarily after optimization. Equation (3) does not explicitly enforce zero correction when the delay is zero.

Questions for authors
  1. Could the authors clarify how ground-truth annotations are aligned with intermediate delay timestamps and reconcile the reported inference latency with the effective playback delays?

  2. How is the identity mapping at zero delay guaranteed after training?

  3. Can the authors better distinguish the proposed feature-level correction from existing latency-compensation approaches, ideally through comparisons with learned, capacity-matched alternatives?

1 comment

We thank the reviewer; both concerns about soundness are well-founded. After receiving the reviews, we audited the evaluation code against the manuscript and removed every claim unsupported by the protocol. The response contains corrections, mechanisms that can be verified in our implementation, and existing measurements.

Corrected protocol. We report results at three playback speeds. At 1×/2×/4×, the annotated nuScenes keyframe timeline is replayed s times faster. Thus, consecutive keyframes are 0.5/s s apart in playback time (0.5/0.25/0.125 s). Playback compresses the timeline but does not interpolate labels; every scored label is an actual annotated keyframe. Each model runs at its measured latency (per-frame logs; 156.8 ms mean for SparseDrive-S). We pair its output with the first annotated keyframe slot strictly after the prediction is ready. The revision removes the Δ = t_small + s·dt delay model, the "≈0.1/0.2/0.5 s effective delay" mapping, Fig. 1(a), Table 9 (dense delay sweep), Sec. 5.7, and the appendix delay-generalization study because the protocol does not support those delay interpretations.

W1/Q1 (ground truth at intermediate timestamps; 156.8 ms vs. claimed effective delays). Both inconsistencies are real. The corrected protocol resolves the second one. The 156.8 ms figure is SparseDrive-S's measured mean latency. At 1×, annotated keyframes are 0.5 s apart in playback time, so this latency does not cross a slot boundary; the prediction is paired with the next keyframe. Faster playback shortens the interval between slots to 0.25 s at 2× and 0.125 s at 4×. The same latency therefore spans more slots and yields greater staleness. The three playback speeds control this realized staleness, not a nominal delay assigned to each condition. We never synthesize ground truth: training targets and evaluation labels are real keyframe annotations, ego-frame changes use recorded odometry, and agent motion is not interpolated.

W3/Q2 (identity at zero delay after training). Agreed: zero-initialization guarantees an identity mapping only at the start of training, so the paper's claim that identity is "preserved by construction" was too strong. At dt = 0, constant-velocity extrapolation is an exact no-op and the residual branch is bypassed. This is why the offline rows match SparseDrive-S exactly. The revision will state this explicitly and make the identity architectural by gating the residual with g(dt) = 1 − exp(−dt/τ). Since g(0) = 0 exactly, the identity at dt = 0 holds for any training trajectory and any random seed.

W2/Q3 (novelty; LASP; learned output-level comparison). We do not claim that the individual components are new. Our contributions are: (i) correcting latent instance features once, before the frozen task heads, so detection, motion, and planning all receive a consistent correction; and (ii) a multi-task streaming evaluation protocol and its metrics (s-minFDE/s-minADE, s-L2/s-ColRate, and PCS), which prior streaming work focused on detection does not provide. Correcting outputs separately can leave tasks inconsistent: for example, a shifted box may disagree with an unshifted trajectory for the same agent. LASP is not directly comparable in our benchmark because it corrects 2D detection outputs using a different sensor suite and protocol. We position it as a detection-only, output-level special case that our feature-level correction generalizes. Table 5 compares our method with four output- or state-level corrections applied to the same stack. The reviewer is right that a learned, capacity-matched output-level baseline is important: the Table 5 baselines are tracker- or rule-based. The revision therefore adds learned, per-task, delay-conditioned correctors with the same 0.55 M parameter budget, directly testing feature-level versus output-level correction.

What remains valid. All methods were scored under identical pairing rules, so the within-protocol conclusions stand on existing measurements: consistent improvements over SparseDrive-S on all six streaming accuracy metrics at every playback speed, with paired per-scene bootstrap intervals excluding zero over 150 val scenes; transfer to a second backbone (UniAD) with comparable relative gains; and wider advantages under jitter, dropped-frame, and nonuniform-timing conditions — all at 0.55 M parameters and <1 ms overhead. The revision also releases per-frame predictions, per-model latency logs, and configs for every reported condition, so the pairing behavior is externally auditable.

We believe the corrected protocol description, the architectural zero-delay identity, and the added learned baseline address your concerns about soundness and about the evidence supporting the feature-level design; we are grateful for the scrutiny, which caught real errors in our delay accounting.

Reviewer_ingVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the staleness of end-to-end driving outputs: by the time inference finishes, the scene has moved. StreamDrive adds small trained modules (0.55M parameters, "<1 ms") to a frozen SparseDrive-S (85.9M parameters). They correct each object's latent instance feature once, after the backbone and before the frozen detection, motion and planning heads.

Critical strengths
  • Latency makes every output of an E2E stack stale, and scoring detection, motion forecasting and planning under one delay protocol is a useful framing that others could adopt.
  • The correction acts after the image cross-attention and before the frozen heads, so no delay-shifted query is fed back into image attention and no backbone retraining is needed.
  • StreamDrive is ahead of SparseDrive-S on every streaming metric of Tables 1–3 and ahead of every post-hoc baseline in Table 5.
Critical weaknesses
  • The model ingests "the latest captured keyframe" and is scored against "the next real nuScenes keyframe" whose timestamp matches the actuation time, and "no labels are interpolated". nuScenes keyframes are annotated at 2 Hz, so keyframe-to-keyframe scoring can only realise offsets in multiples of 0.5 s, yet results are reported at 0.05, 0.1, 0.15, 0.2 and 0.3 s (Fig. 1, Table 9). The cited ASAP benchmark built higher-rate nuScenes labels for exactly this purpose (ASAP); the paper neither uses them nor explains how it obtains sub-0.5 s offsets without them.
  • Following Li et al. (2020), a streaming metric scores each model at its own latency, but Sec. 4 computes the metrics "under simulated delay" and reports a latency only for SparseDrive-S. Two observations suggest one shared delay for all methods: the three baseline detectors in Table 1 keep the same share of their offline mAP (0.29, 0.27 and 0.11 at 1×, 2× and 4×), and AD-MLP, which needs almost no compute, loses as much planning accuracy at 1× (L2 +41%) as SparseDrive-S (+40.5%). Slower models such as UniAD are then not penalised for their own latency, and "playback speed" becomes a label for a fixed offset.
  • SparseDrive-S already loses 70% of its offline mAP at 0.05 s (0.418 → 0.126, Table 9), where the mean ego displacement is 0.28 m (p95 0.65 m), and then only another 0.011 up to 0.2 s, where the displacement has quadrupled. nuScenes mAP matches boxes by centre distance at 0.5, 1, 2 and 4 m (nuScenes), so a 0.28 m shift should mainly affect the tightest threshold. A step at the first nonzero delay followed by a plateau looks like a fixed step of the protocol (input frame, scored frame or t_small) rather than an effect of elapsed time, and Fig. 1(a) starts at 0.05 s, so the step is never drawn. If part of the loss is a protocol artefact, the 70.8% and 88.8% losses that motivate the paper are overstated, and StreamDrive's +0.013 mAP at 1× recovers about 4% of a gap whose cause is unexplained.
  • Zero-initialising the last FiLM layer makes the residual vanish for every dt at the start of training; that is all ControlNet's zero convolutions guarantee (ControlNet). After 20 epochs nothing in Eq. 3 forces γ(0) = β(0) = 0: the sinusoidal code at dt = 0 is not zero (its cosine half equals 1), the scene and ego inputs are not zero, and the CV feature at dt = 0 is MLP_cv applied to a zero vector, i.e. its bias. The offline rows of Tables 1–3, identical to SparseDrive-S in every reported digit (0.418/0.525, 0.992/0.623, 0.61/0.08), therefore suggest that the module is switched off at dt = 0, not that zero-init "confirms" anything. If so, contribution (1)'s "zero offline degradation" holds because the backbone is frozen and the module is bypassed, not "by construction", and should be stated that way. The "− Zero-init" row of Table 4 is reported at 1× only, so it cannot test the offline claim. A residual multiplied by a gate with g(0) = 0 would make the property structural.
Questions for authors
  • For each method, playback speed and sweep point, what are dt, t_small, Δ, the input frame (keyframe or 12 Hz sweep) and the timestamp of the label actually scored? How do 0.05–0.3 s offsets obtain real labels when keyframes are 0.5 s apart, and is each baseline scored at its own latency or at a shared simulated delay?
  • Which value does FiLM receive at test time (realised latency, nominal condition delay, an estimate, or Δ), do the per-instance offsets Δt_i enter Eq. 3, and which value is fed under the jitter conditions of Table 8? If the realised latency is used, how is it known before the prediction exists?
  • How are agent positions, velocities, future trajectories and plans at t + dt obtained for a continuous dt ∈ [0, 0.5] s from 2 Hz keyframes, and can you report streaming detection separately for static and moving objects?
  • How were the streaming rows of UniAD, VAD-Base, AD-MLP and StreamPETR produced, and why does each planner's streaming L2 scale with its offline L2 by a common factor? Which AD-MLP variant gives L2 4.00 and collision 5.89? Can you release per-frame predictions for all methods?
1 comment

We thank the reviewer for the careful reading. Your reconstruction of our protocol is correct. Because annotations are provided at 2 Hz and predictions are paired with the next keyframe slot, realized label offsets occur in 0.5 s increments. The protocol therefore cannot realize the paper's sub-0.5 s delay conditions. We audited the evaluation code against the manuscript and removed every claim unsupported by the protocol.

Corrected protocol (W1). We report results at three playback speeds. The annotated keyframe timeline is replayed 1×/2×/4× faster, giving playback intervals of 0.5/0.25/0.125 s. Playback compresses time but does not interpolate labels. Every scored label is an annotated keyframe. Each model runs at its measured latency, and its output is paired with the first annotated slot after it is ready. Realized staleness is therefore k·0.5 s in dataset time. The revision removes the Δ = t_small + s·dt model (t_small = 0 in our implementation), the "≈0.1/0.2/0.5 s effective delay" mapping, Fig. 1(a), Table 9, Sec. 5.7, and the appendix delay-generalization study. The sweep's compensation horizon did not match the realized slot, so it did not test distinct delays. Finer offsets require higher-rate labels, as in ASAP. The revision clarifies this limitation and cites ASAP as a path forward.

W2 (shared delay vs. own latency). Each run used its model's measured latency. But next-slot pairing quantizes staleness: models finishing within the same slot interval are assigned to the same +1 slot. This explains the nearly constant offline-mAP share across detectors (~0.29 at 1×/2×; ~0.11 at 4×) and similar L2 gains for AD-MLP (+41%) and SparseDrive-S (+40.5%). Sec. 4 now describes a controlled comparison under this shared pairing rule, not Li et al.'s per-model-latency metric, and states the quantization explicitly. We will publish realized-lag distributions derived from the pairing rule and per-frame latency logs.

W3 (step-then-plateau; 70.8%/88.8%). Agreed: the 0.418-to-0.126 drop reflects a one-slot shift (~0.5 s staleness), not a 0.05 s delay; the plateau through 0.2 s is the identified artifact. The main-table drops are valid (0.418 to 0.122 at 1× and 0.047 at 4×). We will report them and recompute recovery using realized staleness.

W4 (zero-init). Agreed: identical offline results come from the dt = 0 path, not zero-initialization. At dt = 0, constant-velocity extrapolation is a no-op and the residual is bypassed; zero-initialization only sets the correction to zero at training start. We restate contribution (1) as "frozen backbone + exact bypass at dt = 0." The revision gates the residual with g(dt) = 1 − exp(−dt/τ), where g(0) = 0, making this property architectural. The "−Zero-init" row is relabeled as a 1×-only training-dynamics ablation, not a test of the offline claim.

Q1. Input: latest keyframe. We remove t_small = 0. Inference uses measured model latency in playback time; the label is the first annotated keyframe after the prediction is ready. The realized offset is k·0.5 s, so sub-0.5 s rows are removed. Baselines use their own latency under the same pairing rule.

Q2. FiLM receives the physical-time extrapolation horizon for the assigned slot, set by the deterministic placement rule rather than wall-clock completion time. Per-instance Δt_i from the timestamp-aware queue uses the same conditioning path. Under Table 8's jitter, the residual uses each frame's jittered delay.

Q3. Agent states at non-keyframe times are never constructed, in training or evaluation. The displacement statistics were analytic and tied to the removed grid (deleted). A static/moving split is not yet computed; committed in the revision from existing per-scene arrays.

Q4. UniAD, VAD-Base, AD-MLP, and StreamPETR used the same wrapper at each model's measured latency. W2 explains their common scaling factor. AD-MLP is the ego-status MLP of Zhai et al. (2023), using its official checkpoint with our adapter. We will clarify its L2 aggregation and release configs and per-frame predictions for all methods.

Because all methods use identical pairing rules, within-protocol comparisons remain valid: paired-bootstrap intervals exclude zero for all six metrics at every speed. This review also helped us correct the paper's claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.