- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StreamDrive, a streaming perception framework that compensates for inference delay by extrapolating latent instance features in end-to-end autonomous driving. It combines a constant-velocity prior with a learned residual conditioned on scene context and ego state. The paper also introduces a multi-task streaming evaluation protocol and the Plan Consistency Score (PCS). Experiments show that StreamDrive improves perception, motion forecasting, and planning performance under streaming inference.
Critical strengths
-
The proposed streaming feature compensation is simple and effective, addressing inference delay across perception, motion forecasting, and planning within a unified framework.
-
The method achieves consistent improvements across multiple tasks while introducing limited additional parameters and inference overhead.
-
The experimental evaluation is reasonably comprehensive, including ablation studies, comparisons with alternative methods, and tests under different latency conditions. The results support the effectiveness and practical applicability of the proposed approach.
Critical weaknesses
-
It is unclear how the training targets are constructed. The model is trained with delays sampled from [0, 0.5] s, but nuScenes provides standard 3D box annotations at 2 Hz. For a delay of 200 or 400 ms, where does the corresponding future ground truth come from? The paper says that labels are not interpolated. Transforming the existing boxes into a future ego frame would account for ego motion, but would not update the positions of moving objects. The same question applies to the fine-grained delays used in evaluation. I could not work out the actual input–target pairing from the current description.
-
The zero-delay behavior needs clarification. Zero initialization makes the residual zero at the start of training, but it does not ensure that it stays zero at
dt=0after training. Is there a bypass or an explicit constraint for this case? The matching offline scores are reassuring, but they do not explain how the claimed exact preservation of the original outputs is achieved. -
The experiments are comprehensive, but the discussion mostly repeats the numbers in the tables. A deeper analysis of the observed trends would help explain the method's behavior. Also, qualitative examples are missing. A few detection visualizations before and after compensation would help show what the module corrects and where it still struggles.
Questions for authors
Please address the questions raised in Weaknesses.
We thank the reviewer for the constructive assessment; both technical concerns are valid. After receiving the reviews, we audited the evaluation code against the manuscript and removed every claim unsupported by the protocol. The response contains corrections, mechanisms that can be verified in our implementation, and existing measurements.
Corrected protocol (context for all answers). We report three playback speeds. Replaying the nuScenes keyframe timeline at 1×/2×/4× gives playback intervals of 0.5/0.25/0.125 s. Playback does not interpolate labels; every scored label is an annotated keyframe. Each model runs at measured latency, and its output is paired with the first annotated keyframe after it is ready. The revision removes the "≈0.1/0.2/0.5 s effective delay" mapping, the Δ = t_small + s·dt model, Fig. 1(a), Table 9, Sec. 5.7, and the appendix delay-generalization study. Keyframe-only scoring cannot realize sub-0.5 s offsets; those results did not measure the claimed delays.
W1 (training-target and fine-grained-delay construction). Your interpretation is largely consistent with the implementation; the revision now describes it precisely. During training, each sample is paired with a real future keyframe, 0.5 s later. Agent boxes and trajectories come from that keyframe's annotations, so moving agents are represented by real future labels rather than by transforming current boxes. The future ego frame comes from recorded odometry. Each sample's delay is the actual elapsed time to its paired keyframe. It is discrete and determined by the keyframe spacing, not sampled continuously as "dt ~ U[0, 0.5] s"; we have corrected that statement in the paper. Evaluation follows the same no-synthesis rule: a prediction from keyframe i is scored against the first annotated keyframe strictly after it is ready, accounting for the model's measured latency. Realized offsets are therefore quantized to 0.5 s increments. The reviewer is right that ego-motion transforms alone would not move agents; this is why agent states come from real future annotations.
W2 (zero-delay behavior after training). Agreed: zero-initialization guarantees a zero residual only at the start of training, so "preserved exactly by construction" was too strong. The offline rows are identical because the dt = 0 path bypasses the correction: constant-velocity extrapolation is an exact no-op at dt = 0, and the FiLM branch is not applied. The revision will state this explicitly and also gate the residual with g(dt) = 1 − exp(−dt/τ). Since g(0) = 0 exactly, the architecture guarantees identity at dt = 0 for any training trajectory and any random seed; it does not depend on initialization or on the bypass convention.
W3 (deeper analysis; qualitative examples). These are fair requests. The revision will add: (i) a static-versus-moving split for streaming detection and motion, computed from existing per-scene arrays, to show where the correction has the greatest effect; (ii) an analysis of accelerating and turning agents, where the constant-velocity prior is less accurate and the learned residual should matter most, including failure cases; (iii) before-and-after examples under latency for detection, forecasting, and planning, showing failures such as occluded or abruptly maneuvering agents as well as successes; and (iv) discussion that interprets the results rather than restating the tables. For example, we will discuss why detection gains plateau in absolute mAP while planning gains grow with playback speed (visibility limits versus accumulating ego displacement), and why motion gains are smaller (the forecast head already models some temporal information). Two existing analyses are also relevant: paired per-scene bootstrap intervals over 150 validation scenes, and the irregular-timing study, where StreamDrive's advantage grows when dropped frames violate frame-indexed assumptions.
What remains valid. All methods were evaluated using the same pairing rules. Existing measurements therefore support the within-protocol conclusions: improvements over SparseDrive-S on all six streaming accuracy metrics at every playback speed, with paired-bootstrap intervals excluding zero; transfer to a second backbone (UniAD); and graceful degradation under jitter, dropped frames, and nonuniform timing. These results use 0.55 M parameters and add less than 1 ms of overhead. The revision will release per-frame predictions, per-model latency logs, and configurations so each reported condition can be reproduced.
We believe the corrected target/protocol description, the architectural zero-delay identity, and the added analyses address your three weaknesses directly; thank you for the careful and collegial review.