acceptodds
Under review as a conference paper at ICLR 2027

Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset

Abstract

Existing event stream based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term, large-scale frame-event visual object tracking dataset, termed FELT. It contains 1,044 long-term videos that involve 1.9 million RGB frames and event stream pairs, 60 different target objects, and 14 challenging attributes. To build a solid benchmark, we retrain and evaluate 23 baseline trackers on our dataset for future work to compare. In addition, we propose a novel Associative Memory Transformer based RGB-Event long-term visual tracker, termed AMTTrack. It follows a one-stream tracking framework and aggregates the multi-scale RGB/event template and search tokens effectively via the Hopfield retrieval layer. The framework also embodies another aspect of associative memory by maintaining dynamic template representations through an associative memory update scheme, which addresses the appearance variation in long-term tracking. Extensive experiments on FELT, FE108, VisEvent, and COESOT datasets fully validated the effectiveness of our proposed tracker. Both the dataset and source code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Term, Visual Object Tracking, Event Cameras, Associative Memory, Benchmark Dataset

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bhn3OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript addresses long-term RGB-Event single object tracking, a problem that has not been systematically evaluated in existing benchmarks. The topic is well motivated, and the amount of work is substantial.

Critical strengths

1.Solid dataset and benchmark contribution. The authors construct FELT, a relatively large-scale long-term RGB-Event tracking dataset, which is meaningful. 2.The method is reasonably connected to the long-term tracking setting. This design of AMTTrack is relevant to several key issues in long-term tracking, such as appearance variation, use of historical information, and template maintenance. The experimental results also show performance gains. 3. The manuscript is well-structured and clearly written, making it easy to follow.

Critical weaknesses
  1. In AMAH, features from preceding layers are used as stored patterns, and later-layer features are used for retrieval. The design itself is reasonable, but the motivation for the selected layer combination and memory bank construction is not fully discussed. The authors may explain why the current layers are chosen and what role shallow or intermediate features play in RGB-Event tracking.

  2. The notation and description of the Hopfield layer could be made more consistent. Both AMAH and ATU use Hopfield layers, but their input meanings and memory sources are different: AMAH is mainly used for cross-layer and cross-modal retrieval, while ATU is used for dynamic template enhancement. The authors should more clearly distinguish the inputs, outputs, and roles of these two modules.

Questions for authors

1.FELT includes long-term tracking attributes such as out-of-view and full occlusion, but the annotation and evaluation rules for these frames are not fully clear. For example, it is not explicitly stated whether bounding boxes are provided when the target is completely invisible, whether such frames are included in SR/PR/NPR computation, and how predictions are judged in absent frames. I suggest the authors clarify these details.

1 comment
AuthorsOpenReview ↗

Response to Reviewer bhn3

We thank the reviewer for the positive assessment and constructive suggestions. We address each point below.

W1: Motivation of the layer combination and memory bank construction in AMAH.

The design follows the complementary roles of shallow and deep features in RGB-Event tracking. Shallow and middle blocks preserve high-frequency structural cues, especially the edges and motion contours captured by event streams, while deep blocks encode semantic identity that remains stable under appearance changes. AMAH therefore uses the features of preceding layers as the Hopfield memory bank for later-layer retrieval: the search-region features at layers 5 / 8 / 11 retrieve against the search-region features of layers {1, 3} / {4, 6} / {7, 9}, so that each retrieval layer recalls patterns of a comparable abstraction level, and the pairing deepens progressively with network depth. Moreover, the retrieval is performed cross-modally (RGB search features retrieve event-side patterns and vice versa), injecting complementary modality evidence at every depth rather than only at fusion. We will add this motivation discussion to the revision.

W2: Notation consistency of the Hopfield layers in AMAH and ATU.

We agree and will fix this in the revision. Specifically, we will (i) unify the notation of the Hopfield retrieval layer across both modules, and (ii) add a comparison table that explicitly distinguishes their inputs, outputs, and memory sources:

ModuleState patterns (input)Stored patterns (memory)Output
AMAHsearch-region features at layers 5/8/11opposite-modality search-region features at layers {1,3}/{4,6}/{7,9}retrieval-augmented search-region features
ATU (template update)dynamic template features sampled from the long-/short-term template memorylearnable prototypesenhanced dynamic templates

Q1: Annotation and evaluation rules for out-of-view / fully occluded frames.

We clarify the rules defined in our benchmark. (1) When the target is completely invisible (out of view or fully occluded), the ground-truth annotation marks the frame as absent with a non-positive bounding box (width or height ≤0\leq 0); we additionally provide a frame-level absence flag file for each sequence. (2) SR / PR / NPR are the three widely used tracking metrics, computed under the standard one-pass evaluation (OPE) protocol as in recent tracking benchmarks (e.g., LaSOT, VisEvent, COESOT): SR is the area under the success curve (overlap thresholds from 0 to 1), PR is the percentage of frames whose center error is below 20 pixels, and NPR is the normalized precision, where the center error is normalized by the target size and reported at a threshold of 0.2. All metrics are computed over entire sequences: absent frames are excluded from the scoring yet remain in the denominator, so they always count as failures, and a tracker is continuously penalized until it re-acquires the target; predictions in absent frames cannot accidentally score, and the re-acquisition speed directly affects the final metrics. We will also release the frame-level absence annotations together with the dataset.

Reviewer_2WArOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The author provides a dataset. The author proposes the FELT dataset, which contains 1,044 video segments, approximately 1.95 million pairs of RGB frames and corresponding event data, 60 types of targets, and 14 challenge attributes, and re-trains and evaluates 23 baseline methods.

  2. The method design is coherent in general. In terms of the method, the author proposes AMTTrack: introducing cross-layer and cross-modal Hopfield feature retrieval in a unified Transformer backbone, and combining short-term and long-term template libraries as well as template reconstruction based on learnable prototypes to improve multimodal representation and target appearance adaptation.

Critical strengths

The dataset performed well under the benchmark settings, and the method design was generally reasonable.

Critical weaknesses
  1. The verification of long-term tracking ability is insufficient. The paper emphasizes the situation where the target reappears after disappearing, but does not clearly explain how to retrieve the target after losing it, and lacks direct experiments to verify the recovery ability.

  2. The contribution of the new module is not yet clear enough. The ablation of key modules was not conducted under the core dataset FELT and the final training configuration, so it is difficult to determine how much each module contributes to the final performance separately.

Questions for authors
  1. The paper has cited and compared MamTrack, CVPR 2025 in FE108. However, it seems that when I looked at this work, the VisEvent score was SR 61.6 and PR 79.2, both of which were higher than the 60.1 and 78.1 of this paper. Can you explain this?

  2. What was the result when using the final training configuration on FELT and removing the feature retrieval and template update modules respectively?

  3. How does the method find the target again after it leaves the screen? Has there been any experiment to verify the success rate of recovering the target?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2WAr

We thank the reviewer for the careful reading and constructive comments. We address each point below.

[W1 & Q3] Direct verification of the recovery ability.

We first clarify the mechanism: AMTTrack performs local search (the search region is cropped around the previous target location, following the one-stream paradigm) without an explicit global re-detection module. Re-acquisition relies on two memory designs: (1) the long-term (LT) template memory stores diverse historical templates via the appearance-diversity-based update mechanism, so when the target re-enters the search region with a changed appearance, the model can re-associate early appearance patterns rather than only the latest, possibly corrupted template; (2) the confidence-gated short-term (ST) update prevents occluded or absent frames from contaminating the memory.

To verify the recovery ability directly, we evaluate on the Out-of-View (OV) and Full Occlusion (FOC) subsets of the FELT test set (197 and 106 of the 314 sequences, respectively). Under our protocol, scores are computed over entire sequences: absent frames remain in the denominator and always count as failures, so a tracker is continuously penalized until it re-acquires the target; these subsets therefore directly measure re-acquisition ability. Comparison with OSTrack retrained by us on FELT:

SubsetTrackerSRPRNPR
OV (197 seqs)OSTrack43.756.954.2
OV (197 seqs)AMTTrack45.157.555.4
FOC (106 seqs)OSTrack33.943.839.6
FOC (106 seqs)AMTTrack35.244.641.2

AMTTrack outperforms the retrained OSTrack on both subsets across all metrics (+1.4/+0.6/+1.2 SR/PR/NPR on OV, +1.3/+0.8/+1.6 on FOC), indicating more robust behavior in disappearance- and occlusion-dominated scenarios. We also note that the absolute scores remain low for all trackers; recovery after long-term absence is far from solved. We will discuss this limitation in the revision and consider a global re-detection branch (e.g., periodic global matching against the LT memory) as future work.

[W2 & Q2] Module-wise ablation on FELT under the final training configuration.

Following the reviewer's suggestion, we have conducted the module-wise ablation on FELT under the final configuration (initialized from the same OSTrack pre-trained weights and trained with the same fixed random seed as the full model), removing the associative feature retrieval (AMAH) and the memory-based template update (ATU) individually:

#ATUAMAHSRPRNPR
154.067.164.6
2✓54.467.365.2
3✓54.767.865.4
4✓✓54.867.965.7

Both modules contribute: removing either one degrades performance (SR 54.8 to 54.7 / 54.4). AMAH contributes more individually (+0.7 SR / +0.7 PR / +0.8 NPR) than ATU (+0.4 / +0.2 / +0.6), and their combination yields the best SR/PR/NPR with +0.8 SR, showing that the two modules are complementary rather than redundant. This table will be added to the revision.

[Q1] Comparison with MamTrack on VisEvent.

We confirm the reviewer's observation: MamTrack (CVPR 2025) reports SR 61.6 / PR 79.2 on VisEvent, higher than ours (60.1 / 78.1). We will add MamTrack to the VisEvent comparison table in the revision using its reported numbers. Three clarifications: (1) the comparison across the two public datasets is mixed: on FE108, MamTrack achieves higher SR (66.4 vs. 65.6), while our PR (95.9) is the best among all methods (MamTrack: 94.2); the gap on VisEvent is also small (≤\leq 1.5 points). (2) AMTTrack is positioned as a general memory-augmentation framework: with the same initialization and training configuration as the baseline, it consistently improves OSTrack on FELT, COESOT, and VisEvent, achieving the best SR/PR on COESOT (68.8 / 82.9) and the best NPR on FELT (65.7). (3) MamTrack has not released usable code (the official repository contains only a README), so it cannot be retrained on FELT and included in the benchmark; we will supplement its unified evaluation on FELT as soon as the code is available. We do not avoid the fact that recent methods lead on individual datasets; such methods are complementary to our benchmark, where they can be evaluated under the unified protocol.

Thank the authors for the detailed rebuttal and additional experiments. The additional ablation studies on FELT and the clarification for MamTrack have addressed some of my concerns. However, regarding target recovery, it remains insufficiently resolved since there is no direct evidence showing whether the tracker can reacquire a target after it disappears and reappears. Moreover, even though the ablation studies show the contributions of both modules, the improvement is relatively modest. Therefore, I would like to keep my score unchanged.

Reviewer_7JhuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The current research on "RGB image + event camera" target tracking mainly relies on short video datasets. It is unclear how the algorithm performs in real scenarios such as long-term occlusion, target leaving and re-entering the frame, and continuous changes in appearance. Therefore, the authors established a long-term tracking dataset FELT and proposed AMTTrack, by using modern Hopfield networks to retrieve and fuse relevant features at different levels and between two modalities, while maintaining long-term and short-term target template memory to adapt to changes in target appearance and reduce the influence of incorrect templates. Experiments on four datasets FELT, FE108, COESOT and VisEvent show that these designs can improve the overall performance of the basic tracker, especially achieving better normalized accuracy and positioning accuracy.

Critical strengths
  1. A large-scale long-term RGB-event target tracking dataset named FELT was proposed, covering approximately 1.95 million frames, 60 types of targets, and 14 complex scenarios. 23 trackers were uniformly evaluated, providing a valuable benchmark for subsequent research.
  2. AMTTrack makes use of the complementarity between RGB images and event data. It fuses features at different levels and from different modalities through Hopfield associative memory, and adapts to changes in the target appearance using long-term and short-term template memory.
  3. The experiment covered four datasets, and the ablation experiments also verified the contributions of each module. Meanwhile, the model only increased approximately 9.3% of parameters and 10.7% of computational load, yet still achieved 31 FPS, demonstrating excellent accuracy, robustness and operational efficiency.
Critical weaknesses
  1. The main criterion is that the video should have more than 1000 frames. However, it fails to adequately address the core issues such as the long-term disappearance and reappearance of the target.
  2. If the target is completely blocked or leaves the screen for a long time, the model may have difficulty re-locating the target.
  3. Part of the performance improvement comes from the stronger pre-trained weights of OSTrack, but this cannot be entirely attributed to the associative memory design.
  4. Lack of variance from multiple runs, statistical significance, and more detailed analysis of failure scenarios.
Questions for authors
  1. Could the authors provide statistics on target disappearance and reappearance in FELT, such as the number and duration of out-of-view and full-occlusion episodes? How is performance specifically evaluated on these sequences?
  2. Does AMTTrack perform global re-detection when the target reappears, or does it continue searching within a local region?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 7Jhu

We thank the reviewer for the comments. We address each concern below.

[W1 & Q1] Statistics of target disappearance and reappearance in FELT.

FELT is designed around long-term disappearance rather than video length alone: beyond the >1,000-frame criterion (1,867 frames on average), every sequence is annotated with 14 challenge attributes characterizing disappearance and reappearance. Out-of-View (OV) is the second most frequent attribute in FELT (see Fig. 5(a) in the supplementary material): target disappearance is a dominant challenge of the benchmark, not a by-product of long videos. In the full FELT (1,044 sequences), OV covers 682 sequences and Full Occlusion (FOC) 334; in the 314 test sequences, the two subsets contain 197 (62.7%) and 106 (33.8%) sequences. Each sequence also provides frame-level absence annotations: the test set contains 3,705 absence episodes (11.8 per sequence, 32.4 frames on average); these statistics will be added to the revision.

Under the standard OPE protocol, SR/PR/NPR are computed over entire sequences: absent frames still count toward the denominator and are scored as failures, so a tracker that fails to re-acquire the target keeps accumulating errors. We thus report the OV and FOC subsets (AMTTrack vs. OSTrack retrained by us on FELT):

SubsetTrackerSRPRNPR
OV (197 seqs)OSTrack43.756.954.2
OV (197 seqs)AMTTrack45.157.555.4
FOC (106 seqs)OSTrack33.943.839.6
FOC (106 seqs)AMTTrack35.244.641.2

AMTTrack consistently outperforms the baseline on both subsets across all three metrics; the low absolute scores further confirm that long-term disappearance and full occlusion are the dominant challenges of FELT.

[W2 & Q2] Behavior after full occlusion / long absence.

We clarify that AMTTrack performs local search (the search region is cropped around the previous target location, following the one-stream paradigm) without an explicit global re-detection module. Two designs mitigate the re-localization difficulty: (1) the long-term (LT) template memory stores diverse historical templates via the appearance-diversity-based update mechanism; when the target re-enters the search region with a changed appearance, the model can re-associate early appearance patterns rather than the latest, possibly corrupted template; (2) the confidence-gated short-term (ST) update prevents occluded or absent frames from contaminating the memory. AMTTrack improves over the retrained OSTrack by +1.4/+0.6/+1.2 SR/PR/NPR on OV and +1.3/+0.8/+1.6 on FOC (see above), indicating stronger robustness on disappearance- and occlusion-dominated subsets. We also note that the absolute scores remain low for all trackers; recovery after long-term disappearance is far from solved. We will discuss this limitation in the revision and consider a global re-detection branch (e.g., periodic global matching against the LT memory) as future work.

[W3] Are the gains merely from the stronger OSTrack pre-training?

No. The module effectiveness is verified under two pre-training settings. The component-wise analysis on COESOT with MAE pre-training (Tables 5-6) shows consistent gains when adding ATU and AMAH one by one. The module-wise ablation on FELT under the final configuration (same OSTrack pre-trained weights) shows +0.4/+0.7 SR for ATU/AMAH individually and +0.8 SR in combination (54.0 to 54.8). Together, the two ablations indicate that the gains come from the module designs rather than a specific initialization.

[W4] Variance, statistical significance, and failure analysis.

All trainings and evaluations of AMTTrack share one fixed random seed across datasets and ablation configurations, so the reported numbers are directly comparable. We further conduct per-sequence paired tests between AMTTrack and the retrained OSTrack over the 314 test sequences: AMTTrack performs better on 214/314 sequences (54.8 vs. 52.3 mean per-sequence SR), with Wilcoxon signed-rank p<0.001p < 0.001 (z=6.8z = 6.8) and paired t-test t=3.5t = 3.5, p<0.001p < 0.001: the improvement is systematic rather than driven by a few sequences.

For failure analysis, the worst 10% of sequences by per-sequence SR (31 sequences) show a clear pattern: 30 contain OV and 24 FOC; after a long absence the target reappears outside the local search region and cannot be re-acquired (consistent with W2). All 31 contain ST (small target, median size about 10$\times$8 pixels), where slight localization errors already fail IoU. Low illumination usually co-occurs with other factors (11/31) and rarely dominates failures alone. A typical case is dvSave-2022_10_29_17_56_15 (FOC+OV+ST): the target is absent for 679 frames and not recovered. Complete statistics and qualitative figures will be given in the revision.

  1. FELT primarily defines long-term sequences as videos exceeding 1,000 frames. In the rebuttal, the authors further emphasize that the dataset focuses on target disappearance and reappearance. It remains unclear whether these scenarios are defined and evaluated in sufficient detail to support the paper’s broader claims about long-term tracking. The broad title itself is not the main issue; the concern is whether the claims match the scope of the evaluation.

  2. AMTTrack uses local search and has no explicit global re-detection mechanism. If the target re-enters the camera’s field of view outside the local search region, the tracker may be unable to find it. The authors’ reported failure case involving a long absence illustrates this limitation.

  3. The paper reports the overall performance of an RGB-Event tracker and ablations of its fusion modules, but does not provide a controlled comparison of RGB-only, Event-only, and RGB-Event inputs using the same model and experimental setup. It is therefore difficult to determine how much of the improvement comes from the event data itself.

AuthorsOpenReview ↗

Response to Reviewer 7Jhu

We thank the reviewer for the follow-up. We address each point below.

[P1] Definition and evaluation of the long-term scenarios.

FELT is designed for long-term RGB-Event tracking in diverse challenging scenes. (1) Broad coverage: beyond the >1,000-frame criterion, every sequence is annotated with 14 sequence-level challenge attributes — including fast motion, deformation, partial/full occlusion, scale variation, illumination variation, low illumination, over-exposure, background interference, and small targets, collected in both indoor and outdoor environments — so the challenge space of the benchmark is explicitly defined. (2) Fine-grained annotation of the core long-term challenge: for target disappearance and reappearance — the defining difficulty of long-term tracking — each sequence additionally provides a frame-level absence annotation (present/absent per frame), which serves as the operational definition of disappearance (out-of-view or fully occluded); under our protocol, absent frames remain in the denominator and always count as failures until the tracker re-acquires the target, so the metrics directly measure long-term recovery rather than only short-term accuracy. (3) Scope: the benchmark is disappearance-dense by construction — 79.6% of the 314 test sequences contain at least one absence episode (3,705 episodes, 32.4 frames on average, up to 679 frames) — and the OV/FOC subsets (197/106 sequences) further isolate disappearance-dominated cases, where, as reported in our initial response, all trackers score low. We therefore position FELT as a benchmark that makes long-term failure modes explicitly measurable, and AMTTrack as consistently improving the baseline on it, rather than as solving long-term recovery.

[P2] Local search without global re-detection.

We agree with this reading — it is precisely the limitation we acknowledged. The memory designs help re-acquire the target when it re-enters the search region (possibly with a changed appearance), but cannot recover a target that reappears outside this region, as illustrated by the reported 679-frame failure case.

[P3] RGB-only / Event-only / RGB-Event controlled comparison.

We have now run this controlled comparison on COESOT: the same model (with ATU as the carrier; AMAH is excluded because it operates on cross-modal features and is undefined for single-modality input) trained from the same MAE initialization with the same seed, epochs, and test settings, where the unused modality is removed at the token level so it never enters the network. RGB-only achieves 62.9/77.2/76.2 and Event-only 54.3/66.6/66.3 (SR/PR/NPR), against 64.1/78.7/77.5 for RGB-Event under the identical setup (the "+ATU" row in Table 5). Adding event to RGB improves SR by +1.2, PR by +1.5, and NPR by +1.3 — consistent across all metrics — while Event-only alone retains ~85% of the dual-modal SR, indicating that the event stream contributes genuinely rather than redundantly, and that fusion with RGB remains the strongest configuration.

InputSRPRNPR
RGB-only62.977.276.2
Event-only54.366.666.3
RGB-Event64.178.777.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.