acceptodds
Under review as a conference paper at ICLR 2027

PRED: Predicting Latent Image Features for High-Frequency RGB-Event Object Detection

Abstract

RGB cameras provide rich semantic information, but increasing their frame rate incurs substantial sensing, transmission, and computational costs. Event cameras capture brightness changes with microsecond-level temporal resolution, preserving high-frequency motion while providing limited appearance cues. Their complementary properties make RGB-event fusion promising for high-frequency detection. However, existing asynchronous detectors typically fuse current events with stale image features from the previous RGB frame. This temporal mismatch prevents the fused representation from fully reflecting the current scene and limits detection accuracy between RGB arrivals.We introduce Predictive RGB-Event Detection (PRED), a frame-rate-decoupled framework that predicts the current latent image representation from the previous RGB frame and subsequent events, reducing temporal mismatch before fusion. To provide reliable supervision for this cross-time prediction, we propose two-stage cross-time distillation: a teacher is first trained independently with access to the current RGB frame and then frozen to provide a stable latent target for PRED. We also introduce the High-Frequency Event Dataset (HiFE), constructed through high-fidelity offline rendering and event simulation to provide temporally aligned 500-Hz annotations for millisecond-scale detection evaluation. Experiments on real-world and synthetic event-camera detection benchmarks show that PRED consistently outperforms prior methods across varying query frequencies. In end-to-end wall-clock replay, PRED nearly keeps pace with a 250-Hz offered query stream and delivers predictions at over the rate of prior asynchronous detectors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
event-based vision, multimodal representation learning, object detection, RGB-event fusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1xXrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose a method for improving the accuracy of object detections in RGB video by pairing RGB frames with event camera data. This way there is no "lag" between predictions and the next frame. The authors show significant improvements in detection accuracy this way. The approach works by using events to predict how features extracted from the previous RGB frame would look in the next. To train this the authors use two-stage disdtallation approach, whereby a teacher network has access to the "next" RGB frame and the student has access only to the previous frame and the events. The paper also introduces a synthetic dataset that is used for training and benchmarking the reuslts.

Critical strengths
  • Paper is resonably clear and easy to understand.
  • Qualitative results show large improvements w.r.t compared methods.
  • I appreciate the fact that the authors did not produce AI slop in this paper, which is sadly rather common nowadays.
  • Extensive ablation studies are great,
  • The promised synthetic dataset could be quite useful to others.
Critical weaknesses
  • How is this paper fundamentally different from event camera + RGB temporal super resolution? I image I could just run such a model to upsample RGB frames temporally, then get my predictions on that higher frequency stream.
  • The authors use 5x temporal bins per polarity, like is done in RVT. It’s not really clear how sensitive the proposed latent feature prediction is to this discretization. To me, this is just basically fusing a low framerate RGB camera with a slightly higher framerate event image camera. The authors say there are two approaches for dealing with events, sparse operations (e.g SNNs) and binning. But there are also approaches for image reconstruction where an image state is maintained and updated on a per event basis (e.g see Continuous-time intensity estimation using event cameras). Why not use a continuous time image reconstruction to generate these features rather than binning? That way the authors would eliminate motion blur and not need to choose an arbitrary binning rate. I think this would be interesting to compare and not too difficult to implement. Also a plot of how performance changes with number of bins.
  • Figure 2 is confusing - the figure shows just two frames and then says it is running at 25x. Not sure what is going on. I think it is not a bad figure but could be clearer about the relationship between vent representation and the operating frequency.
  • The proposed architecture uses an RF-DETR-L DINOv2 image encoder with YOLO26 components. Several compared methods have substantially different backbones, pretraining and training procedures. So the contribution of the temporal feature prediction is somewhat entangled with the benefits of distillation. e.g the no-adapter baseline is evaluated only without frozen distillation. A no-adapter baseline trained with frozen teacher supervision would show whether the improvements come form the event sensor latent feature prediction rather than the additional supervision signal. Does that make sense?
Questions for authors

Nits:

  • There are some instances of what I call “gratuitous math”, i.e equations that neither help explain, prove or motivate the work. What is the point of such formalism other than to confuse the reader and make the work seem more complex? I would recommend:

    • Remove Eq(1). EDI is not implemented or solved in this paper at all. Just say on one sentence “Events encode intensity changes over time. The high temporal resolution can help update features extracted from an earlier RGB image” or something.
    • Eq 11 is a bit trivial no?
    • The rest is up to the authors, but I don’t think all of that is needed to understand or reproduce.
  • Why do the authors use ESIM as their backbone simulator? Since this paper, there have been significant improvements in video-to-event simulation. It’s fine, but a sentence to motivate the use of this particular simulator would be nice.

4 comments

We thank the reviewer for the constructive feedback. We have added the comparison with continuous-time image reconstruction and the ablations on event binning and distillation, and revised the paper and Figure 2 as suggested below.

1. Difference from event camera + RGB temporal super-resolution (W1). Event-guided temporal super-resolution (e.g., TimeLens) interpolates between the previous and next RGB frames, but the next frame is unavailable at query time. More fundamentally, any pixel-space upsampling must re-run the image encoder at every query, which nearly doubles the per-query latency on a T4 even before counting the reconstruction time TreconT_{\mathrm{recon}}. PRED encodes each RGB frame once and updates representations in feature space with the light event pathway.

Per-query latency on a Tesla T4 (TensorRT FP16, 320×224320\times224).

PipelineRGB encodingQuery pathwayTotal
Pixel upsampling + detector1.7 ms2.0 ms3.7 ms + TreconT_{\mathrm{recon}}
PRED0 ms (cached)2.0 ms2.0 ms

2. Comparison with continuous-time image reconstruction (W1, W2). We ran the suggested comparison: we reconstructed the current frame from the previous frame and events with the cited continuous-time intensity estimation method (complementary filter, tuned on validation), and evaluated it with our frozen current-image teacher and with a student without the adapter trained on reconstructed frames. In both cases, the reconstructed frame brings no clear gain over the previous frame It0I_{t_0}, and all remain below PRED-FD.

Detection on DSEC-Det with complementary-filter reconstructions.

DetectorImage inputmAPInterframe mAP
Frozen teacherPrevious frame It0I_{t_0}52.268.2
Frozen teacherReconstructed frame51.963.4
Student w/o adapterPrevious frame It0I_{t_0}63.770.7
Student w/o adapter (trained on reconstructions)Reconstructed frame63.969.5
PRED-FDPrevious frame It0I_{t_0}69.774.0

3. Sensitivity to temporal event binning (W2). The event tensor is not a fixed-rate event frame: each query builds it from all events since the last RGB frame, so queries can be issued at any time. Accuracy is insensitive to the discretization between 3 and 10 bins per polarity and drops slightly only with a single bin, which discards the temporal order of events.

Number of temporal bins per polarity on DSEC-Det (PRED-FD).

Bins per polaritymAPInterframe mAP
168.873.3
369.974.5
5 (default)69.774.0
1070.174.5

4. Figure 2 (W3). Thank you for pointing this out. The two RGB frames in Figure 2 are consecutive 20-Hz frames, the two event images mark the same time points with the continuous event stream between them, and the ticks in between mark query times, at which the current frame is unavailable. "25×25\times" referred to queries at 500 Hz, 25 times the RGB rate. We have revised the figure to read "queries up to 500 Hz" and noted in the caption that each query bins all events since the last RGB frame, so the query rate is independent of both the RGB rate and the event representation.

5. Feature prediction vs. extra supervision (W4). Yes, this makes sense; thank you for suggesting this control, which makes the role of each component clearer. We trained the student without the adapter under the same frozen-teacher supervision (now in Table 5). Supervision alone brings some improvement, while adding the adapter for feature prediction increases the gain substantially.

Adapter vs. frozen-teacher distillation on DSEC-Det.

AdapterFrozen distillationmAPInterframe mAP
×\times×\times63.770.7
×\times✓\checkmark65.271.3
✓\checkmark✓\checkmark (PRED-FD)69.774.0

6. Equations (Nits). We agree. We have removed Eq. (1), keeping the EDI motivation as one sentence in the problem formulation, and replaced Eq. (11) with a one-sentence definition of recovery.

7. Choice of ESIM (Nits). We follow "Event Quality Score (EQS): Assessing the Realism of Simulated Event Camera Streams via Distance in Latent Space" (Chanda et al., CVPRW 2025), which compared event simulators against real DSEC data and found ESIM closest to the real events (EQS 0.866 vs. 0.773 for v2e and 0.501 for PIX2NVS). We have added this motivation to Section 4, and we will explore newer video-to-event simulators in future work.

Dear Reviewer 1xXr,

Thank you again for your insightful comments. We have responded to each of your points and included the suggested comparisons in our response and the revised paper. If you have any remaining questions or concerns, we would very much appreciate hearing them and would be happy to discuss further. Thank you for your time and consideration.

Best regards, Authors

Reviewer_1xXrOpenReview ↗

Rebuttal Reply

Dear authors - thank you for your replies. Fair enough - given what I have learned here I would be happy to update my recommendation to a "Weak Accept". Best of luck!

Dear Reviewer 1xXr,

Thank you for your follow-up and for updating your recommendation. Your suggestions have helped improve the paper, and we sincerely appreciate your time.

Reviewer_gaB4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the temporal mismatch between stale RGB features and current event information in high-frequency RGB–event object detection. To tackle this issue, the authors propose PRED, a frame-rate-decoupled detection framework that employs event-guided latent image feature prediction to update image representations at query time without repeatedly performing image encoding, thereby achieving a favorable balance between detection accuracy and inference efficiency. Furthermore, the authors introduce a two-stage cross-time knowledge distillation strategy, in which a teacher model with access to current RGB observations provides feature-level supervision for the student network. The paper also presents HiFE, a synthetic benchmark with native 500 Hz annotations, enabling high-frequency interframe detection evaluation without relying on interpolated annotations. Overall, the paper addresses a well-motivated problem and presents a reasonably novel methodological framework. The experimental results demonstrate substantial improvements in detection accuracy and high-frequency inference efficiency, while the proposed HiFE benchmark provides an additional contribution to the research community. Nevertheless, several aspects warrant further investigation, particularly the generalization capability over longer temporal intervals, robustness under challenging real-world conditions, and the necessity and individual contributions of certain key design choices. Additional experiments and analyses would help strengthen the empirical justification of the proposed method and clarify its broader applicability.

Critical strengths
  1. The core idea is innovative and offers strong potential for high-frequency perception applications. The paper addresses the temporal mismatch between stale RGB features and current event information in high-frequency RGB–event interframe object detection by proposing PRED, a frame-rate-decoupled detection framework. Through event-guided latent image feature prediction, PRED retains cached RGB features as appearance anchors while leveraging subsequent event information to update image representations at query time. This design decouples computationally expensive image encoding from high-frequency detection: the image encoder runs only when a new RGB frame arrives, while feature prediction and detection are performed through a lightweight event-driven pathway between consecutive RGB frames. By mitigating the performance degradation associated with directly reusing stale image features, the framework offers a promising solution for applications such as autonomous driving, where high perception rates and low inference latency are essential.
  2. The training strategy is well designed, with targeted supervision for cross-time feature prediction. The paper introduces a two-stage cross-time distillation strategy. First, a teacher model with access to the current RGB frame is trained independently and then frozen to provide two levels of supervision-image features and fused features-for a student network that observes only the previous RGB frame and intervening event information. Compared with training based solely on detection labels, this strategy provides explicit learning targets for recovering current-time image representations while further encouraging the predicted features to remain effective for subsequent multimodal fusion. Moreover, comparisons with EMA-based self-distillation and ablation studies involving different feature-supervision configurations and event-encoder designs provide empirical evidence for the benefits of fixed teacher targets and specialized event-encoding pathways.
  3. The framework achieves strong detection performance and inference efficiency, supported by encouraging experimental results. On the complete DSEC-Det test set, PRED-FD achieves 69.7 mAP, outperforming FlexEvent by 12.3 percentage points in the reported comparison and approaching the 72.4 mAP achieved by the current-image reference model. In terms of inference efficiency, PRED achieves a cached-query latency of 2.00 ms and a maximum model-only query rate of approximately 498.9 Hz. In end-to-end wall-clock replay experiments conducted on an RTX 4090, PRED achieves 98.1% new-prediction coverage under a 250 Hz offered query rate. These results demonstrate the framework's advantages in detection accuracy, query-level efficiency, and sustained processing capability, highlighting its potential for high-frequency visual perception.
  4. The HiFE benchmark makes a valuable contribution to high-frequency detection evaluation and offers broader research potential. The paper introduces HiFE, a synthetic benchmark constructed through offline rendering in Unreal Engine 5 and event simulation using ESIM. It provides temporally aligned image sequences, event streams, and native annotations at 500 Hz, enabling direct evaluation of detections at intermediate query times without relying exclusively on interpolated bounding boxes between adjacent RGB frames. Compared with evaluation protocols based on interpolated annotations, HiFE provides a more direct basis for assessing detection performance at high temporal resolution. In addition, the availability of depth, optical flow, and instance-segmentation annotations creates opportunities for extending the benchmark to other temporal vision and multimodal perception tasks.
  5. The evaluation is multifaceted and provides substantial empirical support for the proposed framework. Beyond comparing detection performance across methods, the paper investigates the framework from several complementary perspectives, including feature-distance measurements between predicted and current-image representations, feature-error visualizations, performance across different query frequencies, end-to-end wall-clock replay, and ablation studies of feature prediction, teacher supervision, and event-encoder configurations. The feature-substitution experiments examine whether predicted representations retain information useful for downstream detection, while the component ablations investigate the respective effects of different supervision signals and architectural choices. Collectively, these experiments provide a broad evaluation of representation quality, detection accuracy, temporal generalization, and inference efficiency, strengthening the empirical support for the paper's main conclusions.
Critical weaknesses
  1. The temporal generalization capability and operating boundaries remain insufficiently explored. The model is trained on DSEC-Det using adjacent RGB frames separated by approximately 50 ms and evaluated at query intervals ranging from 2 to 50 ms. These experiments demonstrate its ability to accommodate different query times within a single RGB frame interval. However, its performance beyond the training interval remains unclear, particularly under low-frame-rate RGB input or prolonged RGB unavailability caused by consecutive frame drops. Although the paper reports detection results at multiple query intervals, it does not systematically characterize how feature prediction errors and detection performance evolve over longer temporal gaps. Evaluating intervals beyond the training range and analyzing the relationship between temporal gaps, feature prediction errors, and detection accuracy would help establish the operating boundaries of the proposed method.
  2. The realism of the synthetic benchmark and its cross-domain generalization remain insufficiently validated. HiFE is constructed using Unreal Engine 5 scene rendering and ESIM event simulation, providing native annotations at high temporal resolution. Nevertheless, simulated event streams may exhibit distributional discrepancies with real sensor measurements due to differences in sensor noise, pixel response non-uniformity, and imaging conditions in complex environments. Although the paper evaluates the method on both the real-world DSEC-Det dataset and the synthetic HiFE benchmark, these experiments do not establish whether a model trained on synthetic data can generalize effectively to real-world scenarios. The effects of simulation parameters and scene variations on model performance also remain insufficiently characterized. Cross-domain evaluation, particularly training on HiFE and testing on real-world data, would provide stronger evidence of the benchmark's practical utility and the method's generalization capability.
  3. The necessity and computational trade-offs of the dual-event-encoder architecture require further investigation. PRED employs separate event encoders for temporal feature adaptation and downstream detection. The paper compares this design with a shared event encoder and demonstrates the advantages of dedicated encoders under the evaluated settings. However, alternative lightweight architectures, such as a shared backbone with separate output heads, have not been systematically investigated. Moreover, the trade-offs among parameter count, computational complexity, inference latency, and detection accuracy remain insufficiently quantified. Although the existing ablation studies support the effectiveness of dedicated event encoders, further comparisons would help establish whether the additional architectural complexity is justified relative to more computationally efficient alternatives.
  4. Robustness to realistic perturbations in practical perception systems remains insufficiently assessed. Although PRED is evaluated on real-world driving data, the paper does not systematically investigate its robustness to several perturbations that may arise in practical deployment, including temporal misalignment between RGB frames and event streams, event sensor noise, and substantial illumination or exposure changes. Such perturbations may disrupt the correspondence between event features and cached image features, potentially degrading latent image feature prediction and subsequent detection performance. Controlled experiments involving varying degrees of temporal misalignment, sensor noise, and imaging changes would help characterize performance degradation under these conditions and provide a more comprehensive assessment of the framework's robustness in challenging real-world environments.
  5. The fairness and comparability of inference-efficiency evaluations could be further improved. The paper compares PRED with several baseline methods in terms of detection accuracy and inference latency. However, the methods differ in input resolution, hardware platform, numerical precision, execution backend, and inference scheduling strategy, which may complicate direct comparisons of runtime performance. For example, PRED is evaluated using TensorRT FP16, whereas DAGr's graph neural network module runs in PyTorch FP32. In addition, DAGr is evaluated using batched inference at query times, while its native incremental execution mode is not benchmarked. These implementation and evaluation differences may influence the reported latency and throughput. Where feasible, additional comparisons under standardized hardware, input resolution, precision, and execution settings—or sensitivity analyses across deployment configurations-would help distinguish the benefits of the proposed model architecture from those arising from implementation and evaluation choices.
Questions for authors
  1. What is the effective temporal horizon for feature prediction under prolonged RGB unavailability? The paper avoids accumulating prediction errors by preventing predicted features from being written back to the cache. However, when RGB frames are unavailable for an extended period, such as during consecutive frame drops or under lower RGB input frame rates, how does the accuracy of predictions over longer temporal intervals deteriorate? Is there a maximum effective prediction horizon beyond which the reliability of feature prediction significantly degrades? The authors are encouraged to quantify the relationship between temporal gap length, feature prediction error, and downstream detection performance to better characterize the operating range and limitations of the proposed method.
  2. How are the weights of the two distillation objectives determined, and what are their respective contributions? The proposed two-stage distillation strategy supervises both image features and fused features. How are the weights assigned to the two distillation losses, and how sensitive is the model's performance to these weights? The paper already compares image-only distillation with joint image-and-fusion distillation. Could the authors further clarify the respective contributions of the two supervision objectives, particularly the incremental benefit of fusion-level supervision over image-level supervision? Additionally, have alternative loss-weight configurations been explored, and if so, how do they affect performance?
  3. How were the ESIM simulation parameters for HiFE selected, and to what extent does the simulation capture real event-camera characteristics? HiFE uses ESIM to generate event streams with specified contrast thresholds, a refractory period, and a low-pass time constant. What motivated the selection of these parameters, and were they calibrated against measurements from real event cameras? Furthermore, does the simulation account for realistic sensor noise characteristics, such as dark-current-related noise and pixel response non-uniformity? Further analysis of the sensitivity of detection performance to simulation parameters and sensor noise would help clarify the realism and limitations of the proposed benchmark.
  4. What accounts for the advantage of frozen-teacher distillation over EMA-based self-distillation? The paper reports that the frozen-teacher strategy outperforms EMA-based self-distillation. Beyond the stability provided by fixed supervision targets, what factors account for this performance gap? For instance, could differences in teacher feature quality, target consistency, or the use of current-frame RGB information contribute to the observed results? Further analysis would help clarify the underlying reasons for the effectiveness of the frozen-teacher strategy and establish a more precise understanding of its advantages over EMA-based self-distillation.
  5. Can the proposed feature prediction framework be extended to other vision tasks? The paper primarily evaluates the proposed feature prediction framework on object detection. Could the event-guided latent image feature prediction mechanism also be applied to other vision tasks, such as semantic segmentation and depth estimation? If so, which components could be reused directly, and which would require task-specific modifications, particularly with respect to prediction targets, distillation objectives, and downstream task heads? Further discussion or preliminary experiments would help clarify the generalizability and broader applicability of the proposed framework.
  6. What is the computational cost of RGB feature extraction, and could it become a system-level bottleneck? The paper reports end-to-end performance with RGB inputs at 20 Hz and high-frequency detection queries. However, the runtime of the image encoder itself is not fully characterized. How long does it take to extract features from a single RGB frame, and how does this cost affect overall system throughput when image encoding runs concurrently with event-driven queries? Under high query loads or substantial computational resource contention, could image encoding or cache updates become a system-level bottleneck? Reporting per-frame image encoding latency and its contribution to end-to-end processing would provide a more comprehensive assessment of the framework's practical efficiency.
3 comments

We thank the reviewer for the insightful comments. We have conducted additional experiments on longer temporal horizons, depth transfer, and perturbation robustness, and provided further analyses of our design and simulation choices below.

1. Performance over longer RGB gaps (W1, Q1). Feature prediction remains effective up to about 100 ms (twice the 50-ms training interval). As the RGB gap increases up to 400 ms (up to 7 consecutive dropped frames), feature prediction error and detection mAP degrade smoothly together without abrupt failure: at 100 ms, the predicted feature error still stays below the unadapted cached error at 50 ms and mAP remains at 64.8, whereas from 200 ms residual error grows larger and accuracy drops more noticeably.

Feature error (foreground cosine distance ×100\times100 to the teacher's fused features) and mAP across RGB gaps on DSEC-Det.

RGB gapFeature error (cached →\rightarrow predicted)mAP
50 ms2.4 →\rightarrow 0.869.7
100 ms5.7 →\rightarrow 1.964.8
200 ms10.9 →\rightarrow 4.552.0
400 ms16.5 →\rightarrow 9.432.6

2. Extension to other tasks (Q5). Yes. We transferred PRED to metric depth estimation on DSEC at a 200-ms RGB gap. The frame-rate-decoupled architecture and frozen-teacher distillation of image and fused features are reused directly, while the image encoder (Depth Anything V2-S), event-encoder initialization (YOLO26n-Depth), depth head, and training schedule are task-specific. As in detection, both the adapter and distillation improve accuracy, and PRED nearly matches the current-image teacher.

Metric depth estimation on DSEC at a 200-ms RGB gap.

ModelAbsRel ↓\downarrowδ1\delta_1 ↑\uparrow
Teacher, current frame0.1090.893
PRED0.1090.888
PRED w/o distillation0.1150.880
PRED w/o adapter and distillation0.1360.834

3. Synthetic-to-real generalization (W2). To test whether models trained on our synthetic data transfer to real data, we trained the same RGB–event depth network under identical settings on data from HiFE's rendering pipeline and on EventScape, a widely used synthetic event-depth dataset, and evaluated both zero-shot on DSEC and MVSEC. The model trained with our pipeline transfers consistently better on both real datasets.

Zero-shot synthetic-to-real relative depth transfer on DSEC and MVSEC.

Training dataDSEC AbsRel ↓\downarrowDSEC δ1\delta_1 ↑\uparrowMVSEC AbsRel ↓\downarrowMVSEC δ1\delta_1 ↑\uparrow
EventScape (synthetic)0.1780.7510.4390.446
HiFE's rendering pipeline (synthetic)0.1600.7910.3500.530

4. Dual event encoders (W3). Following the reviewer's suggestion, we trained a variant with a shared event backbone and separate output heads for the adapter and detection under the same supervision. Separate heads save minimal compute and latency over dedicated encoders while still losing 2.7 mAP, whereas dedicated encoders bring a clear accuracy gain for a modest cost increase.

Event-encoder designs on DSEC-Det (cached-query GFLOPs and T4 TensorRT FP16 latency at 320×224320\times224).

Event encoderParamsGFLOPsLatency (ms)mAPInterframe mAP
Shared35.8M2.632.8465.272.6
Shared backbone, separate heads36.4M2.813.1367.073.2
Dedicated (PRED-FD)37.0M2.923.3069.774.0

5. Robustness to misalignment, sensor noise, and illumination changes (W4). We added controlled experiments for the three perturbation types the reviewer listed, evaluating PRED-FD and the current-image teacher on DSEC-Det: temporal misalignment, event sensor noise, and RGB illumination changes. Temporal misalignment and sensor noise cause negligible degradation, and under severe exposure and gamma changes PRED-FD drops by nearly the same margin as the current-image teacher, showing that the drop comes from the RGB encoder rather than a broken correspondence between events and cached features.

Detection accuracy (mAP) under temporal, sensor-noise, and illumination perturbations on DSEC-Det.

PerturbationPRED-FDTeacher (current frame)
None69.772.4
Event timestamp shift (−20-20 ms)68.6 (−1.1-1.1)72.4 (0.0)
Event sensor noise (+20% random events)69.5 (−0.2-0.2)72.3 (−0.1-0.1)
Under-exposure (RGB gain ×0.25\times0.25)64.9 (−4.8-4.8)67.9 (−4.5-4.5)
Over-exposure (RGB gain ×4\times4)66.0 (−3.7-3.7)68.6 (−3.8-3.8)
Gamma compression (RGB γ=0.5\gamma = 0.5)68.9 (−0.8-0.8)71.8 (−0.6-0.6)
Gamma expansion (RGB γ=2\gamma = 2)57.5 (−12.2-12.2)60.1 (−12.3-12.3)

6. Latency under matched settings (W5). We re-measured the batch-1 cached-query latency of all methods on the same devices at matched resolution (320×224320\times224 for PRED and FlexEvent, 320×215320\times215 for DAGr) across all supported backends, and PRED remains the fastest in every setting. Because DAGr's graph changes with every event window, it cannot use static CUDA Graphs or TensorRT; moreover, its incremental mode takes 35.4 ms per event on the T4 (whereas events arrive every 0.77 μ\mus), so batched inference is its fastest mode.

Cached-query latency (ms) under matched settings; PyTorch FP32 reports eager (CUDA Graph).

MethodT4 PyTorch FP32T4 TensorRT FP16Orin NX PyTorch FP32Orin NX TensorRT FP16
PRED20.3 (5.6)2.061.4 (27.6)9.9
FlexEvent26.3 (12.7)4.261.6 (54.6)20.9
DAGr, batched73.1–655.2–
DAGr, incremental (per event)35.4–103.0–

7. Distillation weights (Q2). The default weights (500 for image, 6000 for fusion) were chosen empirically during early development and kept fixed afterwards. Either loss alone improves substantially over the undistilled baseline, and combining both brings a further gain. Moreover, halving or doubling either weight changes mAP and interframe mAP by at most 0.3 points, showing that accuracy is insensitive to the exact weights.

Ablation and sensitivity of image and fusion distillation weights on DSEC-Det.

Image weightFusion weightmAPInterframe mAP
0063.471.0
500068.472.6
0600069.173.6
5006000 (default)69.774.0
250600069.674.2
1000600069.874.0
500300069.774.0
5001200069.974.3

8. ESIM parameters (W2, Q3). In HiFE, we set the ESIM contrast threshold (0.4) to match the average event density of real DSEC data (40.8 vs. 44.3 events/pixel/s) and set the refractory period (0.2 ms) and low-pass time constant (8 ms) close to standard simulator presets (v2e's noisy preset: 0.5 ms and 5.3 ms). Regenerating the HiFE test events across variations of these three parameters shows that PRED-FD is insensitive to the contrast threshold and refractory period, with only a minor drop at a larger low-pass time constant. The default HiFE split does not inject sensor noise or pixel non-uniformity, which we will provide as configurable options in the released pipeline.

Sensitivity of PRED-FD (mAP) on HiFE to ESIM simulation parameters.

ESIM setting20 Hz500 Hz
Default (threshold 0.4, refractory 0.2 ms, low-pass 8 ms)29.535.5
Contrast threshold = 0.329.535.8
Contrast threshold = 0.529.235.3
Refractory period = 0 ms29.535.5
Refractory period = 0.5 ms29.535.5
Low-pass time constant = 0 ms29.036.1
Low-pass time constant = 16 ms27.735.0

9. Frozen teacher vs. EMA (Q4). Since both the EMA and frozen teachers receive the current frame It1I_{t_1}, the gap comes from target consistency and teacher quality rather than access to It1I_{t_1} alone. Under EMA, the student's predictions converge toward the evolving EMA target while both drift away from the fixed current-image reference (0.214 →\rightarrow 0.431 in Table 4, vs. 0.049 under frozen supervision). Moreover, as a moving average of the student, the EMA teacher remains bounded by the student's quality: even when fed It1I_{t_1} directly, it reaches only 58.3 mAP, compared with 72.4 mAP for the independently trained frozen teacher (Tables 4 and 1).

10. Cost of RGB encoding (Q6). Image encoding does not become a bottleneck. On a Tesla T4 (TensorRT FP16), encoding one RGB frame takes 1.7 ms (vs. 2.0 ms per cached query), occupying only about 3% of GPU time at 20 Hz. In our wall-clock replay, queries do not wait for it: RGB encoding runs in a separate worker while each query reads the latest completed cache, and any contention between the two workers is already reflected in the reported coverage (98.1% at 250 Hz).

Dear Reviewer gaB4,

Thank you again for your detailed and constructive feedback. Following your suggestions, we have added a number of new experiments and clarifications in our response and the revised paper. Should any point still be unclear, we would be grateful if you could let us know, and we would be happy to elaborate. We sincerely appreciate your time.

Best regards, Authors

Reviewer_T6dDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PRED detects objects between RGB frames by combining an RGB camera with an event camera. Existing detectors fuse current events with image features from the last RGB frame. PRED instead uses the events since that frame to predict the current image features and then fuses the prediction with the events. A teacher trained with the current image supervises the predicted image features and the fused features, and is kept frozen during this training. The image encoder runs only when a new RGB frame arrives, while a small event pathway answers every query. The paper also introduces HiFE, a synthetic driving dataset with native 500-Hz annotations, and reports accuracy on DSEC-Det and HiFE together with model latency and wall-clock throughput.

Critical strengths
  1. The paper addresses a concrete problem with a practical design. Each query starts from the latest cached RGB features, so prediction errors do not accumulate across queries, and the current image is needed only during training. The ablations are informative. With the student architecture fixed, the frozen teacher raises complete-test mAP from 63.4 to 69.7, and the comparison of image-only and joint targets, the feature substitution, and the foreground feature distances show where this gain comes from.

  2. The evaluation separates settings that are easy to confuse. HiFE provides native annotations between RGB frames, the paper reports the filtered interframe intervals and the complete DSEC-Det test set separately, and the wall-clock replay reports dropped requests together with output rate and latency. This makes the speed claim more credible than model-only timing would.

Critical weaknesses
  1. The ablations do not yet show how much of the gain comes from predicting the current time. The frozen teacher clearly helps, but part of the improvement may come from distillation from a stronger network in general, a question the authors leave open in Appendix F.3. FAOD (supplementary Section 7) already trained its alignment module with an MSE target from temporally aligned RGB features and found no benefit. The paper should therefore explain why its target works here, for example because of the independently trained frozen teacher or the joint image and fusion targets. A direct control would keep the teacher checkpoint and all training settings fixed and compute the image targets from the previous frame I_{t0} instead of the current frame I_{t1}. The difference between the two runs would measure the value of the cross-time target, which the EMA comparison cannot show. On HiFE, DAGr is the only baseline and the test set is a single trajectory of about 40 s. Reporting the current-image teacher and the student without the adapter (as in the first row of Table 12) would show how much of the 21-point gain over DAGr comes from feature prediction.

  2. Some comparisons may not use the same inputs or evaluation intervals. First, Table 1 does not state whether the frame-only YOLO26x and RF-DETR-L receive I_{t0} or I_{t1}. If they receive I_{t1}, the 6.6-point gain over YOLO26x compares two different settings. Second, the FlexEvent row in Table 2 is copied from its paper, and its 20-Hz entry equals its complete-test score (57.4). The 20-Hz entries of PRED-FD and DAGr instead differ from their complete-test scores (71.9 vs. 69.7 and 44.1 vs. 41.9), so this row may cover different intervals, and the authors should confirm or re-evaluate it. Third, accuracy is measured at 768×512, while the replay runs at 320×224 and reports no mAP, so the accuracy of the fast configuration is unknown. The replay also compares TensorRT FP16 PRED with batched PyTorch FP32 DAGr instead of native incremental DAGr. Please report mAP for the timed configuration and limit the abstract's claim of "over 5× the rate of the prior competitive asynchronous detectors" to the tested DAGr deployment.

Questions for authors
  1. How does the frozen cross-time target differ from the aligned-feature target that did not help FAOD? Can a control with previous-frame versus current-frame targets quantify the benefit?
  2. Which RGB frame do the frame-only baselines in Table 1 receive at t1? Do all rows of Table 2, including FlexEvent, use the same target timestamps, annotations, and filtered intervals?
  3. What mAP does PRED reach at 320×224 under the replay schedule, and does each query use only RGB features whose timestamp precedes the query? Can the teacher and the student without the adapter be reported on HiFE?
4 comments

We thank the reviewer for the thorough evaluation and valuable suggestions. We have added the cross-time target control and HiFE baselines, and clarified the protocols and runtime settings below.

1. Value of the cross-time target (W1, Q1). Our teacher is not a larger network: it shares the student's no-adapter architecture and differs only in receiving the current frame It1I_{t_1} rather than It0I_{t_0}. As suggested, we fed this frozen teacher It0I_{t_0} under identical settings (Table 5). Supervising with It0I_{t_0} matches the undistilled model, whereas It1I_{t_1} improves mAP by 6.2 points over this control, showing that the gain comes from the cross-time target.

Previous- vs. current-frame distillation targets on DSEC-Det.

Distillation targetmAPInterframe mAP
None63.471.0
Frozen teacher (It0I_{t_0})63.570.8
Frozen teacher (It1I_{t_1}, PRED-FD)69.774.0

2. Difference from FAOD (W1, Q1). Our distillation differs from FAOD's in target source and supervised features. First, FAOD regresses its Align Module output to the aligned frame's features, both from the network being trained (its Supp. Eq. 4), whereas our targets come from an independently trained, frozen teacher: an evolving EMA target drops mAP to 57.1, below the undistilled model (63.4), while the frozen teacher reaches 69.7 (Table 5). Second, beyond FAOD's shallow RGB feature supervision, we supervise deep multi-scale image features and fused features, raising mAP from 68.4 (image-only) to 69.7 (Table 15).

3. HiFE baselines and test set scale (W1, Q3). We added two baselines on HiFE under the same recipe (Table 14): the current-image teacher and the no-adapter student. Of the 21.0-point average gain over DAGr, the no-adapter student accounts for 16.9 points and feature prediction adds 4.1, recovering over half of the 7.7-point gap to the teacher. We agree that the single-trajectory test set is limited: HiFE targets native 500-Hz labels unavailable in real datasets (40 s, 20,022 frames, 314,220 boxes), and we will release its automated pipeline to render more trajectories without manual labeling.

Detection mAP on HiFE across query frequencies (Hz).

Method2036456090180500Avg.
DAGr9.812.012.713.413.714.214.412.9
PRED-FD w/o adapter24.128.329.530.631.232.432.529.8
PRED-FD29.533.033.934.735.235.735.533.9
Teacher, current frame37.537.637.737.737.737.637.037.5

4. Input of the frame-only baselines (W2, Q2). YOLO26x and RF-DETR-L receive It0I_{t_0}, the same image as PRED, and are trained and evaluated against the labels at t1t_1. The 6.6-point gain over YOLO26x is thus measured under the same setting.

5. Evaluation protocol of FlexEvent (W2, Q2). The FlexEvent row was cited from its paper, and with its code now available, we found that it uses its own protocol (stricter box filtering; 9,130 of 12,128 frames scored), whereas all other rows use the official DAGr protocol. We therefore re-evaluated all three methods under both protocols: all three score higher under FlexEvent's protocol, while PRED-FD remains ahead under both.

DSEC-Det mAP under both evaluation protocols.

MethodDAGr protocolFlexEvent protocol
DAGr41.947.9
FlexEvent (released weights)52.558.2$^\dagger$
PRED-FD69.771.9

†^\daggerReported in FlexEvent's repository (its paper reports 57.4); reproduced by our pipeline.

6. Timed configuration (W2, Q3). At 320×224320\times224 on the complete test set, PRED (PRED-FD) reaches 55.4 (56.2) mAP under the DAGr protocol and 61.4 (62.1) under FlexEvent's. In our wall-clock replay, each query uses only past inputs: the latest encoded RGB features and the events since that frame.

7. Abstract claim (W2). We agree that the abstract’s speed claim should be supported by comparisons under matched runtime settings. We therefore benchmarked DAGr, FlexEvent, and PRED on the same devices and revised the abstract’s final statement to “…while running over 2×2\times faster than prior RGB–event detectors.” Moreover, batched PyTorch FP32 is already DAGr's fastest mode: its per-event mode takes 35.4 ms per event on a T4 (vs. 0.77 μ\mus between events), and its dynamic-graph GNN cannot use static CUDA Graphs or TensorRT.

Cached-query latency (ms); PyTorch FP32 reports eager (CUDA Graph).

MethodT4 PyTorch FP32T4 TensorRT FP16Orin NX PyTorch FP32Orin NX TensorRT FP16
PRED20.3 (5.6)2.061.4 (27.6)9.9
FlexEvent26.3 (12.7)4.261.6 (54.6)20.9
DAGr, batched73.1–655.2–
DAGr, incremental (per event)35.4–103.0–

Dear Reviewer T6dD,

Thank you again for your thoughtful review. We have carefully addressed your comments and added further clarifications and experimental results in our response and the revised paper. If anything remains unclear, we would very much welcome your questions and would be glad to provide further details. We look forward to hearing your thoughts.

Best regards, Authors

Thanks for the detailed response. The control with previous-frame targets answers my main concerns. The protocol clarifications and the HiFE reference models also resolve my doubts. I will keep my original positive score.

Dear Reviewer T6dD,

Thank you for your follow-up and for your positive assessment. We are glad that the additional control and clarifications have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.