acceptodds
Under review as a conference paper at ICLR 2027

SPAR: Monocular Sparse 3D Point Tracking with Events

Abstract

Continuous 3D point tracking is fundamental to dynamic scene understanding. Monocular event streams provide sparse and intermittent image-plane observations, making it challenging to maintain query identity and recover metric motion without direct longitudinal measurements. We propose SPAR, a sparse 3D point tracker that uses only monocular events after image-based initialization. Its query-conditioned interval–geometric matching combines observed-event support with sampled pre-trigger response history through Hit–Survival and refines correspondences with reliability-weighted local motion. An interval-evidence marginal objective further incorporates this candidate-conditioned history into correspondence learning. For 3D inference, we introduce a Query-conditioned Anchor-Relative Trajectory head (QART) that separates one-time metric anchoring from event-driven relative motion and carries correspondence confidence, support reliability, and local geometry into temporal 3D inference. We construct EventTraj3D from BlinkVision, comprising 13.630 billion asynchronous events and 6,494 annotated query trajectories. Experiments show lower 2D and 3D errors on EventTraj3D and the lowest RMSE among compared methods on E-3DTrack, while initialization perturbations show reduced sensitivity to depth-prior errors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event-based Vision, 3D Point Tracking, Monocular Event Cameras, Sparse Point Tracking, Event-based 3D Tracking

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_wD1gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a novel framework for point tracking in using monocular event cameras. Specifically, their major contributions are the use of pre-trigger interval evidence to filter between potential candidate point tracks, learning to lift tracks to 3D with the use of initial depth estimation and local group motion and introducing a new benchmark based on the BlinkVision dataset called EventTraj3D, which includes RGB frames, 2D and 3D metric tracks as well as being balanced across categories. They show improvements in both 2D and 3D tracking against a number of event based point trackers.

Critical strengths

The following are considered strengths of the paper:

  • Evaluation performance supports the efficacy of their method. The authors show state-of-the-art performance on their introduced benchmark for both 2D and 3D tracking. They also show good results on the existing E-3DTrack benchmark, coming first on 5/7 metrics and second the remaining 2 by a small margin.
  • The model variant ablations further support the novel contribution of including pre-trigger evidence for filtering tracks.
  • The shuffled history is a good addition to the ablations to ensure that the temporal history is a relevant contribution.
Critical weaknesses

The following are considered to be weaknesses of the paper:

  • The paper is extremely difficult to understand. The abstract and introduction reference lots of field-specific phrases without context (e.g. candidate-evaluation gap, pre-trigger brightness histories, symmetric thresholds, longitudinal motion). The first figure attempts to motivate the problem but it is not properly labelled or referenced well in the text so simply adds to the confusion. The related works section does not attempt to resolve any of these ambiguities. Lastly, the method section is incredibly difficult to follow. It lacks any motivation for each module, so the reader is simply thrust into equations without any understanding of why they are needed, figures 2 and 3 are meant to aid in the understanding but instead cause more confusion due to a lack of explanation of symbols and link back to the text and even the additional method explanations in the supplementary do not fix the confusion.
  • Although the new EventTraj3D benchmark does have its merits (e.g. large number of training and evaluation sequences, 2D and 3D tracks alongside event data, ...), the details of how this dataset is created and the contribution over the original BlinkVision is not clear from the text. This needs to be expanded to fully understand the contribution of the dataset.
  • There are not enough details on what the "Direct XYZ" variant is in the ablation table. The text says that it "retain encoders, GRU and input features" but this is not clear. What is aim of including this in the ablation? This should be clear.
  • The candidate matcher is missing an explanation in the paper. There does not seem to be any details about the architecture, how many candidates are predicted and how these are chosen. The same issue follows for the "Event Backbone" referenced in figure 2 which is not mentioned anywhere in the text or explained in architecture terms.
  • Although the "Initial-depth sensitivity" sub-section is a welcome ablation, the writing lets it down so it is unclear what the authors are trying to conclude from the experiment. Furthermore, the comparison to the "Direct head" is also not clear. What type of model is this? What architecture does it have? Why is the comparison against it important?
  • Again, although the "Hit-survival analysis" has good intentions and the shuffled history is a good experiment, it is just very poorly explained and therefore the affect of including it is then lost on the reader. A lot of the details are in the supplementary so when reading the main paper we have no context for what we are looking at. The figure in panel a seems to have a box and black point, either of which could be the query but both are together are confusing so makes the reader worry the figure may be AI generated. Lastly, it only uses a set of hand-picked examples and not the whole test set so seems limited.
Questions for authors
  1. What architecture is used for the candidate matcher?
  2. What architecture is used for the event backbone? And how does this compare to other models in the evaluation?
  3. What contributions have been made on top of BlinkVision to construct EventTraj3D? And what are the length of the clips in number of frames?
  4. Can we have more details on the "Direct XYZ" variant in the ablation table?
  5. Can we have more details on the "direct head" model from the ablation and why this is used as a comparison?
  6. How does the method perform on large point displacements? Are these captured by the evaluation benchmarks? If not, is there a good reason for avoiding this?
6 comments

W1. Why are Hit–Survival, local geometry, and QART needed?

The candidate-evaluation gap arises when similar observed-event support masks incompatible pre-trigger histories (Introduction, lines 45–75; Fig. 1). Symmetric thresholds are an idealized event-model assumption (App. A.1), and longitudinal motion denotes motion along the viewing ray (App. F.1). Hit–Survival evaluates candidate histories, local geometry reweights motion-consistent supports, and QART retains correspondence reliability and local motion for 3D inference (Secs. 3.2–3.3).

The notation in Fig. 2 is defined in App. A.3 (lines 765–776). In the revised PDF, we will define these terms at first use, introduce each module's purpose before its equations, and clarify the symbols, captions, and main-text links for Figs. 1–3.

W2 / Q3. What does EventTraj3D contribute beyond BlinkVision?

We appreciate the reviewer’s recognition of the dataset’s value. Resources combining events, persistent point identities, and metric 3D trajectories remain limited. EventTraj3D builds on BlinkVision simulation to provide monocular tracking data across indoor/outdoor scenes and rigid, articulated, and non-rigid motion, complementing real stereo E-3DTrack (App. C.1, Table C2).

Construction contributions. We tried propagating 2D points by optical flow and sampling depth, but observed unstable correspondences during propagation and near occlusions and depth boundaries. We therefore bind queries to native surface points, bridge absolute correspondences across reference frames, and check overlapping positions and depths to preserve physical identity. Queries must satisfy local event coverage, silence-duration, and competing-location discriminability checks. The primary query determines clip duration; supports span the clip, while local points retain validity masks. Same-instance overlap is limited, and splits are disjoint by physical instance.

Clip length and scale. Table C1 reports 1,427 clips, 6,494 trajectories, and a 999/214/214 split. Complete clips span 18–96 frames. The revised PDF will include two construction algorithms: native-track linking, and event-quality screening with clip selection, together with the selection thresholds.

Li et al. BlinkVision: A Benchmark for Optical Flow, Scene Flow and Point Tracking Estimation using RGB Frames and Events. ECCV 2024. Paper

Li et al. 3D Feature Tracking via Event Camera. CVPR 2024, 18974–18983. Paper

W3 / Q4. What are Direct XYZ and the direct head designed to test?

Direct XYZ and “direct head” denote the same variant. Line 421 introduces “Direct XYZ,” and line 423 immediately refers to it as “The direct head.” App. C.3 (lines 928–931) specifies the same encoder and GRU architectures, input features, initialization, and training protocol, with output units and bounds adapted for direct coordinates.

ComponentArchitecture / comparison
Four input encoders1156 / 775 / 14 / 8 dims → 384 dims each
Fusion and temporal modelReliability-weighted support pooling; Linear(1536,384) → GRUCell(384,384)
Direct XYZ decodingLinear(384,3); direct XYZ in a shared reference scale
QART decodingLinear(384,3) → β·tanh → initial anchor + initial depth × relative offset

The variants are trained separately and differ in output representation: Direct XYZ regresses coordinates at a common scale, whereas QART normalizes displacement by query depth and restores metric position from a fixed anchor (Sec. 3.3, Eq. 7). The comparison therefore tests how 3D position is represented rather than additional network capacity.

Output2D EPE (px)3D EPE (mm)3D FDE (mm)
Direct XYZ3.8861167.571644.48
Full QART3.213400.69821.74

Direct XYZ has comparatively close 2D error but substantially higher 3D EPE/FDE, supporting normalized anchor-relative decoding. The restricted GRU still has higher error with the same relative output, supporting the additional value of candidate and neighborhood information. The revised PDF will use “Direct XYZ (direct head)” consistently and present the architecture and output forms together in App. C.3.

W4 / Q1. Candidate Matcher architecture and proposal selection

The matcher combines current events, initial templates, and temporal context to propose endpoints and base matching scores (Sec. 3.2, lines 216–224). Its structure is summarized below.

ComponentArchitecture
Query and memory1162→768→384 MLP; GRUCell(768,384)
Cross-attentionQ: Linear(384,384); K/V: Linear(389,384)
Base scoresFeature similarity + endpoint appearance + candidate priors/normalization
2D readout769→384→192→2 MLP; posterior mean + residual

Candidates surround the previous estimate: 207 per query and 47 per support (25 local, 13 disk, and 9 recovery). Cross-attention provides base scores; Hit–Survival and local geometry further assess candidates before probability-weighted 2D readout (Eqs. 2–6), allowing temporal and motion evidence to distinguish similar appearances. The revised PDF will place proposal rules and architecture details beside the matcher description.

W4 / Q2. Event Backbone architecture and inputs

The backbone is the event encoder in Sec. 3.1 (lines 197–203). Five temporal bins and two polarities form a 10-channel volume; C(a,b,s) denotes a 3×3 convolution, eight-group GroupNorm and SiLU, and R(c) a two-convolution residual block. The architecture follows.

ComponentArchitecture
Event backboneC(10,96,1) → R(96) → C(96,192,2) → R(192) → C(192,384,2) → R(384) → R(384) → GN(8) → SiLU
Event feature mapB×384×(H/4)×(W/4)
Reference template31×31 grayscale patch → C(1,192,1) → C(192,192,2) → R(192) → C(192,384,2) → mean pooling → 384-D
Initialization encoderSame architecture as event backbone; independent weights

The backbone supplies matching features, reference patches anchor appearance, and per-pixel trigger order supports candidate-history tests. DeepEvT combines local matching with cross-track attention; ETAP uses a spatiotemporal Transformer. Our convolutional encoder and candidate cross-attention feed history and local-geometry assessment (Secs. 2.1, 3.1–3.2). The revised PDF will consolidate architecture and algorithm details.

Messikommer et al. Data-Driven Feature Tracking for Event Cameras. CVPR 2023, 5642–5651. Paper

Hamann et al. ETAP: Event-based Tracking of Any Point. CVPR 2025, 27186–27196. Paper

W5 / Q5. What does the direct-head depth-sensitivity test show?

“Direct head” is Table 4’s Direct XYZ: line 421 introduces the variant and line 423 immediately uses this name; App. C.3 specifies the architectural controls. Table 4 compares unperturbed accuracy. App. F.2 instead fixes trained weights, applies identical initial-depth perturbations with three paired seeds, and subtracts each model’s unperturbed error. At 10% and 30% noise, added 3D EPE is 272.72/377.77 mm and 904.26/1207.96 mm, respectively (Ours/Direct XYZ; Table F1), supporting lower initialization sensitivity.

With anchor-relative decoding and GRU capacity held fixed, full QART lowers 3D EPE from 488.84 to 400.69 mm relative to the restricted-input GRU, separately supporting the value of candidate and neighborhood information for 3D accuracy. The paired perturbation results show lower initialization sensitivity for the combined pipeline, not independence from the initial metric anchor (Eq. 8; App. F.1–F.2). The revised PDF will clarify these distinct findings and use the name 'Direct XYZ (direct head)' consistently.

W6. Why these Fig. 5 controls, and what do the markers mean?

History analysis and tracking gains. Fixing weights, candidates, and base scores isolates temporal association within the same match (lines 453–461; App. D). Cases (i–ii) move probability toward better locations without changing the mode; case (iii) corrects the selection, reducing mode error from 14.519 to 1.950 px (Table D1). Each case has 64 valid shuffles preserving locations, polarities, event counts, and the multiset of interval durations. True history gives higher P₂ and lower posterior-weighted error than endpoint-only scoring and the shuffled mean in all three cases (Apps. D.1–D.2). P₂ is probability mass within 2 px of ground truth.

On the full test set, removing Survival scoring raises 2D EPE from 3.213 to 3.830 px and also increases 3D errors. Removing the interval objective likewise reduces accuracy. Candidate history thus supports more accurate final trajectories beyond the local cases. The revised PDF will distinguish the purposes of fixed-candidate controls and retrained ablations and connect their settings to the reported results.

Figure markers. The box and dot represent the same query at different times and therefore share the Query identity label. The hollow box marks the initial location; the dark dot marks the previous estimate and the common start of paths to current candidates A and B; the star marks the current ground truth. The revised PDF will explain each marker in the Fig. 5(a) caption and add explicit temporal indices.

Q6. Can SPAR track large accumulated displacements?

Large cumulative displacement primarily challenges long-term point identity and drift control rather than single-step matching. The high temporal resolution of event cameras enables short observation intervals and incremental tracking, as also demonstrated by EKLT (IJCV 2020) and DeepEvT (CVPR 2023). However, small matching errors can accumulate over time, potentially causing trajectory drift or incorrect reassociation with similar neighboring points.

SPAR addresses these challenges through candidate-conditioned pre-trigger evidence and reliability-weighted local geometry, which help preserve point identity and suppress accumulated matching errors (Sec. 3.2, Eqs. 2–6). On EventTraj3D, 70.09% of test trajectories exhibit cumulative displacements of at least 25 px, indicating that substantial accumulated motion is common in this benchmark and directly relevant to continuous event-based tracking. SPAR achieves a 2D EPE of 3.71 px on this subset, only 0.50 px higher than its 3.213 px EPE on the full test set. This limited degradation indicates that SPAR maintains reliable point correspondences under substantial accumulated motion, consistent with its design for mitigating identity switches and cumulative tracking drift.

Gehrig et al. EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames. IJCV 128, 601–618 (2020). DOI: 10.1007/s11263-019-01209-w

Messikommer et al. Data-Driven Feature Tracking for Event Cameras. CVPR 2023, 5642–5651. Paper

Reviewer_JqLiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the problem of 2D and 3D point tracking using an initial reference image and monocular event streams, with an image-derived depth prior for initialization. It proposes an event-specific tracking architecture and learning objectives, with the main distinguishing factors being correspondence matching that exploits both observed events and the absence of expected events, local geometric constraints, and a recurrent head that directly predicts 3D trajectories relative to the initial depth estimate. The method is evaluated on a newly constructed benchmark and an adapted public dataset, outperforming the corresponding adapted baselines.

Critical strengths
  • The proposed hit–survival matching criterion and associated training objective are potentially interesting sensor-specific contributions.

  • The experiments seem to establish basic advantages of the approach compared to alternatives under the considered input configuration.

Critical weaknesses

(in order of appearance)

  • Readability. The abstract is very dense with terminology (SPAR, Hit–Survival, interval-evidence marginal objective, Query-conditioned Anchor-Relative Trajectory), which is hard to understand and parse prior to reading the actual paper. In that sense, the abstract mostly consists of names that do not intuitively explain the underlying ideas. The main text itself appears to be heavily polished for conciseness, sharpness, and brevity, but achieves the opposite effect: its flow is unnatural and hard to process. The text almost seems written to be read by another machine.

  • Method exposition. The method is particularly hard to follow right from the start. For example, it is not intuitively clear what is meant by "image positions" and "metric positions" (line 182). Additionally, the mathematical descriptions often lack explicit dimensions of tensors, vectors, etc., which further complicates the exposition. The paper repeatedly introduces mathematical constructions without first establishing an intuitive picture of what is being computed and why.

  • Experimental setup. Crucially, the rationale behind the experimental setup is unclear. Why is only a single initial RGB frame used, rather than the full RGB stream, as is done in the original TAPFormer, to the best of my understanding? When is this scenario practical, and why was it configured this way? What if the system were periodically reinitialized using new frames? Additionally, the paper does not provide comparisons against RGB-only tracking methods to establish the advantages of using an event-based sensor. To the best of my understanding, the parent BlinkVision dataset supports such an evaluation.

  • Baseline comparisons. For E-3DTrack, the evaluation setup also diverges from the original stereo-event configuration. More generally, it is not entirely clear whether the adapted baselines are evaluated under comparable conditions and appropriately optimized for the considered setup. In particular, TAPFormer originally uses RGB frames throughout tracking, whereas the authors replace these with event-accumulation images. A clearer explanation of the baseline adaptations and their implications for the fairness of the comparison would be helpful.

Questions for authors

Please address the main questions regarding the rationale behind the experimental setup and the consistency of the evaluation with respect to previous methods and the original datasets.

3 comments

W1. How will readability be improved?

Our method addresses two connected challenges: reliable point correspondence under sparse events and 3D motion recovery without direct depth (Introduction, lines 45–84). In the revised PDF, the abstract will explain these ideas before naming the modules. The main text will also introduce each component's intuition before its equations and improve transitions among the motivation, figures, and method.

W2. What do the position variables and tensor dimensions mean?

“Image positions” are 2D pixel coordinates; “metric positions” are metric 3D positions in the initial camera frame (Sec. 3.1, lines 182–185). Apps. C.2 and F.1 define the metrics and viewing-ray ambiguity. The event input is B×10×H×W; features are B×384×(H/4)×(W/4). QART encodes four inputs of 1156/775/14/8 dimensions into 384-D features (App. A.3). The revised PDF will clarify their meanings and dimensions beside the equations.

W3. Why one initial RGB frame, rather than periodic or continuous RGB?

Purpose and practical setting. We study sparse 3D point tracking with one initial RGB reference when frequent frame capture or transmission is undesirable. Monocular events do not directly provide metric depth; the image supplies appearance and depth references, refined by the first four event observations. Templates and the metric anchor then remain fixed, and only events update 3D trajectories (Sec. 3.1; App. A.4). All compared methods share this initialization.

Periodic RGB and RGB-only comparison. Periodic RGB reinitialization adds appearance and depth cues, changing the observation setting. We also compare with continuous-RGB CoTracker3 + Metric3D v2 (ViT-S), which estimates depth at every frame, whereas SPAR obtains metric depth only at initialization.

ConditionMethod3D EPE (mm) ↓3D ADE (mm) ↓3D FDE (mm) ↓
High-speed motion (top 1%, >156.01 px/s)SPAR626.1278.4299.22
RGB-only827.95209.22173.66
Strong brightness variationSPAR311.078144.135159.405
RGB-only486.191285.935499.727

SPAR has lower errors on all three metrics in both settings despite using only one initial depth estimate. Strong brightness changes disrupt RGB appearance matching (RGB-only 3D FDE: 499.727 mm), whereas event-driven tracking is almost unaffected by these brightness variations in this setting and retains useful motion cues. More moderate 3D motion and depth variation in the brightness-change subset also explain its lower absolute 3D EPE than the high-speed subset. RGB and events are complementary: RGB initializes appearance and metric depth, while events support high-temporal-resolution tracking.

Karaev et al. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. ICCV 2025, 6013–6022. Paper

W4. Is the monocular E-3DTrack protocol and TAPFormer adaptation fair?

Monocular protocol. We use E-3DTrack left-camera events, one initial image, and intrinsics; labels are for training and evaluation only. Right-camera events, stereo disparity, and per-step ground-truth depth are excluded. All methods share initialization and are retrained (Sec. 4.1, lines 350–353; App. C.3, lines 920–927).

TAPFormer adaptation. After initialization, its image branch receives corresponding-window event-accumulation images, and its event branch the same-window event representation. Both branches are retained and the method retrained (App. C.3, lines 925–927). The revised PDF will clarify this common monocular-event protocol.

MetricSPAR (Ours)ETAP + DAE-R
RMSE (m) ↓0.21340.2539
[email protected] (%) ↑9.7310.39
[email protected] (%) ↑22.7418.08
[email protected] (%) ↑39.0027.40
[email protected] (%) ↑22.5523.36
[email protected] (%) ↑41.7335.72
[email protected] (%) ↑57.4946.14

Our RMSE is 0.2134 m versus 0.2539 m for ETAP+DAE-R; FA/TFR are best at 0.15/0.2 m and second at 0.1 m (Table 3), supporting overall 3D accuracy and trajectory retention under the common protocol.

Li et al. 3D Feature Tracking via Event Camera. CVPR 2024, 18974–18983. Paper

Liu et al. TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events. CVPR 2026, 37053–37062. Paper

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the discussion period approaches its deadline, we would appreciate it if you could review our rebuttal and let us know whether any concerns remain unresolved. We would be happy to provide further clarification if needed.

Thank you sincerely for your careful review and constructive feedback.

As the discussion period draws to a close, we would greatly appreciate your consideration of our rebuttal and the additional clarifications provided. If these responses have satisfactorily addressed your concerns, we would be grateful if you could update your review and reconsider your rating accordingly.

Thank you again for your valuable time and thoughtful suggestions, which have helped us improve our work.

Reviewer_tmcwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SPAR, a monocular event-based framework for sparse 3D point tracking using only event observations after image-based initialization. It introduces Hit–Survival matching to exploit pre-trigger event history and a query-conditioned anchor-relative trajectory head (QART) for metric 3D estimation. The authors construct EventTraj3D from BlinkVision and demonstrate improved 2D and 3D tracking accuracy on EventTraj3D and E-3DTrack.

Critical strengths
  1. Hit–Survival incorporates pre-trigger event history into candidate matching, providing a meaningful approach to resolving ambiguous correspondences.

  2. SPAR achieves substantial improvements in 2D and 3D tracking accuracy across two datasets, supported by component ablations and depth-prior robustness experiments.

Critical weaknesses
  1. The ablations do not separately quantify the contributions of pre-trigger Survival scoring and the interval-evidence marginal loss across the full test set.

  2. The evaluation omits recent methods such as MATE and E-TraMamba and lacks detailed performance breakdowns for challenging low-event and ambiguous-motion scenarios.

Questions for authors
  1. Can the authors provide separate ablations for pre-trigger Survival scoring and the interval-evidence marginal objective?

  2. Can the authors compare against more recent event-based trackers or clarify why these methods are excluded?

  3. Can the authors report performance on low-event-density, repetitive-pattern, and non-rigid-motion subsets to better assess robustness?

2 comments

W1 / Q1. What do Survival scoring and interval learning each add?

Survival evaluates pre-trigger consistency in candidate scoring (Eqs. 3–4); the interval objective uses this evidence to supervise matching (Eq. 9; App. B.2). We separately disabled each component, retrained the variants, and evaluated the full test set:

Setting2D EPE (px)3D EPE (mm)3D FDE (mm)
Full model3.213400.69821.74
w/o Survival scoring (Hit retained)3.830412.64895.63
w/o Hit4.420423.92910.13
w/o interval objective3.540405.64865.63

Removing Survival scoring or the interval objective raises 2D EPE to 3.830 and 3.540 px, respectively, and also increases 3D errors. History therefore benefits both candidate assessment and learning for final tracking. The revised PDF will state each ablation setting and its results separately.

W2 / Q2. Coverage of MATE and E-TraMamba

MATE. MATE is discussed in Sec. 2.1 (lines 111–117). Without access to its official implementation during our experiments, we independently reimplemented and retrained it. On the same full test set, SPAR has lower 2D EPE and FDE:

Method2D EPE (px)2D FDE (px)
MATE (our reimplementation)5.5297.958
Ours3.2136.345

E-TraMamba. We did not have access to a usable official implementation or the new dataset for this concurrent work during our experiments. Its monocular branch could use event-predicted depth; SPAR directly fuses candidate and neighborhood evidence for trajectory prediction. This relates the proposed combination to the input setting described in Sec. 2.2, lines 131–133.

Q3. How robust is tracking to sparse events, repetitive patterns, and non-rigid motion?

App. C.1 (lines 862–863) identifies intermittent events, repetitive appearance, and non-rigid/articulated motion. We selected corresponding test trajectories and used the same tracks and event inputs for both methods. Sparse-event cases are defined by limited event support around each query (25–50% retention), rather than low scene-wide activity; repetitive appearance requires at least 50% competing-location similarity. Each cell reports Ours / ETAP+DAE-R. The revised PDF will specify how these subset criteria are measured.

ConditionTracks2D EPE (px)2D FDE (px)3D EPE (mm)
Low-event-density108.07 / 12.8217.81 / 27.311274.59 / 1645.64
Repetitive appearance305.66 / 11.359.73 / 19.25988.40 / 1555.71
Non-rigid-motion236.13 / 9.3410.49 / 16.061162.23 / 1376.52

All three groups have lower 2D EPE, 2D FDE, and 3D EPE, supporting effective tracking under sparse local observations, similar appearances, and articulated motion. This is consistent with jointly using candidate histories and reliable local motion (Apps. A.1–A.2).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the discussion period approaches its deadline, we would appreciate it if you could review our rebuttal and let us know whether any concerns remain unresolved. We would be happy to provide further clarification if needed.

Reviewer_V2JzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SPAR addresses monocular event-based sparse 3D point tracking. After initialization using a reference image and a depth prior, it receives only events. Hit–Survival evaluates correspondence candidates, while the trajectory head combines matching confidence and neighborhood motion to predict 3D trajectories. The paper also introduces EventTraj3D and evaluates the method on two datasets.

Critical strengths
  1. The input setting has practical relevance: tracking requires neither subsequent RGB frames, stereo input, nor per-timestep depth. Evaluating whether a candidate implies a plausible pre-trigger history is also well motivated.

  2. The results support the effectiveness of the complete system. On EventTraj3D, 3D EPE decreases from approximately 509 mm for the strongest compared pipeline to 401 mm. SPAR also achieves the lowest RMSE among the compared methods on E-3DTrack.

Critical weaknesses
  1. Removing Hit and Survival together does not isolate the benefit of pre-trigger history.
Hit measures support from observed events, whereas Survival evaluates the plausibility of the preceding history. Table 4 removes both, so the degradation could originate from either component. The current evidence establishes more clearly that the complete module is useful than that the newly introduced pre-trigger history improves final tracking more generally. The fixed-candidate controls in Figure 5 provide encouraging evidence, but mainly demonstrate improvements in candidate distributions across three selected cases, rather than final tracking accuracy. Given that this is the paper’s most distinctive contribution, I would like to see a direct comparison between Hit-only and Hit-plus-Survival using final tracking metrics.

  2. Without a simple temporal 3D head attached to a 2D event tracker, QART’s additional benefit remains unclear.
A natural solution under this input setting is to predict 3D displacement from 2D trajectories, matching confidence, and initial depth using a standard temporal network. The external baselines mainly combine 2D tracking with independent depth estimation, while Direct XYZ retains the full input features and changes the output parameterization. I therefore cannot yet tell how far this straightforward extension would go, or how much additional benefit comes from QART’s candidate information and neighborhood geometry. Since prior work such as DELTA has studied relative initial-depth outputs, this control should also use a comparable relative-scale representation.

  3. Update latency is not reported, leaving practical event-processing efficiency unclear.
High temporal resolution is an important motivation for event cameras. Parameter count and memory usage do not establish processing speed. Please report update latency under specified hardware, query counts, and event input sizes, including the overhead of the added modules.

Questions for authors
  1. MATE (ICCV 2025) already uses motion guidance and historical features to reduce event-matching ambiguity, and is discussed in this paper. Could the authors provide a 2D tracking comparison with MATE to assess whether explicit pre-trigger history offers additional benefits over existing motion and temporal modeling?

  2. For Weakness 2, could the authors keep the same 2D frontend, initialization, normalized output representation, GRU, and 3D supervision, but provide only 2D trajectories, matching confidence, and initial depth as inputs? Comparing this variant with full QART would distinguish the benefit of a standard temporal 3D predictor from that of the proposed information fusion.

  3. I am also interested in combining depth predicted by E2Depth or DepthAnyEvent-R with E-TraMamba’s monocular branch. This pipeline would likewise rely only on event observations after initialization. Any existing results or a brief discussion would help position SPAR relative to an established 3D tracking framework. This is an optional discussion; a full additional experiment is not required.

  4. Do the authors plan to release the implementation, EventTraj3D data, and evaluation protocol? I would welcome the authors’ responses to these concerns, particularly further discussion of the relevant prior work and clarification of the key comparisons. I will adjust my score based on the authors’ responses.

3 comments

W1. Does pre-trigger Survival improve final tracking?

Sec. 3.2 (lines 225–249) and Eqs. 3–4 distinguish observed-event support from pre-trigger scoring. We separately removed Hit, Survival scoring, or the interval objective, retrained each variant, and evaluated final trajectories on all 214 test clips:

Setting2D EPE (px)3D EPE (mm)3D FDE (mm)
Full model3.213400.69821.74
w/o Survival scoring (Hit retained)3.830412.64895.63
w/o Hit4.420423.92910.13
w/o interval objective3.540405.64865.63

Removing Survival while retaining Hit raises 2D EPE from 3.213 to 3.830 px and also increases 3D EPE/FDE. Removing the interval objective likewise degrades accuracy, supporting both candidate scoring and history-guided learning (Eq. 9; App. B.2). With weights, candidates, and base scores fixed, Fig. 5 and App. D show that true history shifts probability toward better-localized candidates and corrects the selection in case (iii). The revised PDF will distinguish scoring from learning and include these separate ablations.

W2 / Q2. Is a standard 2D-track GRU sufficient for 3D?

Both variants use the same 2D frontend architecture, initialization, 384-D GRU, relative output, and 3D supervision, with 2D positions and differences added to both. The restricted variant retains tracks, differences, confidence, and initial depth but masks candidate, support, and geometry features; query references and time intervals remain shared. Both are retrained.

Model3D EPE (mm)3D ADE (mm)3D FDE (mm)
Restricted-input GRU488.84447.23954.88
Full QART400.69376.56821.74

Full QART lowers 3D EPE/ADE/FDE, supporting candidate distributions, support reliability, and local motion as cues beyond 2D tracks (lines 272–276; App. A.3). DELTA has examined depth parameterization for 3D tracking; our GRU comparison retains the same anchor-relative output to isolate information fusion. Direct XYZ instead retains the full input to test output parameterization (lines 421–426; App. C.3). The revised PDF will present the inputs, outputs, and shared training conditions of both controls together.

Ngo et al. DELTA: Dense Efficient Long-range 3D Tracking for Any Video. ICLR 2025. Paper

W3. Update latency and the lightweight accuracy–speed trade-off

All methods use the same RTX 4090 24 GB GPU, sequences, query sets, and event windows; our voxel input is 10×540×960, with query counts varying by sequence. Average latency runs from CPU-ready input to synchronized 3D output on the CPU. It includes initialization windows and all tracking modules, but excludes data loading, voxelization, weight loading, and common initial-depth inference.

Method2D EPE (px)3D EPE (mm)Update (ms)
SPAR (full)3.213400.69104.510
ETAP + DAE-R4.189509.0483.708
ETAP + E2Depth4.189656.0676.141
TAPFormer + DAE-R6.034514.79142.156
TAPFormer + E2Depth6.034660.20132.740
DeepEvT + DAE-R4.604511.5328.828
DeepEvT + E2Depth4.604657.9823.312

Lightweight configuration. The full model has the lowest 2D/3D errors in this comparison, but not the shortest latency. The new lightweight configuration retains the inputs and Hit–Survival, reduces supports from eight to three per query, shares graph computation, and uses 75% of the full sampling budget.

Method2D EPE (px)3D EPE (mm)3D FDE (mm)Update (ms)
SPAR (full)3.213400.69821.74104.510
SPAR (lightweight)3.740448.51908.6767.570
ETAP + DAE-R4.189509.04981.1783.708

The lightweight model takes 67.57 ms versus 83.708 ms for ETAP+DAE-R, with lower 2D and 3D EPE. Reducing supports and sampling thus retains an accuracy–latency advantage over this baseline. The revised PDF will specify the timing boundaries, variable query loads, and lightweight configuration.

Q1. How does SPAR compare with MATE?

We did not have access to a usable official implementation when the experiments were conducted. We independently reimplemented and retrained MATE from the paper and evaluated it on the same full test set.

Method2D EPE (px)2D FDE (px)
MATE (our reimplementation)5.5297.958
Ours3.2136.345

Our method has lower mean and final 2D errors than our MATE reimplementation. MATE uses motion guidance and historical features; SPAR additionally evaluates candidate-implied pre-trigger responses (Sec. 2.1, lines 111–117; App. A.1). The separate Survival ablation also supports the value of this evidence.

Q3. How would E-TraMamba plus event depth compare?

Feasibility and modeling differences. Sec. 2.2 (lines 131–133) describes E-TraMamba’s monocular branch with per-step depth. Supplying that depth from E2Depth or DepthAnyEvent-R could yield a system whose observations after initialization are exclusively events. The proposed combination first estimates per-step depth and then uses it for 3D tracking. SPAR instead fuses candidate correspondences, local motion, and reliability from a single initial depth anchor (Sec. 3.3; Apps. A.3 and C.3). This preserves evidence beyond a 2D coordinate or scalar depth; the restricted-GRU result supports its value.

Comparison availability. E-TraMamba is concurrent arXiv work. We did not have access to a usable official implementation or its EvD-PointOdyssey resources during our experiments, so no protocol-matched result is available. The revised PDF will clarify the input relationship and modeling distinction between this combination and our method.

Q4. Will the code, weights, data, and protocol be released?

The revised PDF will provide detailed architecture specifications, algorithmic procedures, trajectory construction and filtering rules, and evaluation protocols, including the initialization, coordinate conventions, validity criteria, and metric definitions in Apps. C.2–C.3. Upon acceptance, we plan to promptly release the source code, trained weights, EventTraj3D dataset, fixed splits, and data-construction, training, and evaluation scripts to support reproducibility.

Thank you sincerely for your careful review and constructive feedback.

As the discussion period draws to a close, we would greatly appreciate your consideration of our rebuttal and the additional clarifications provided. If these responses have satisfactorily addressed your concerns, we would be grateful if you could update your review and reconsider your rating accordingly.

Thank you again for your valuable time and thoughtful suggestions, which have helped us improve our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments on the method, component contributions, 3D trajectory inference, and experimental setup. We have clarified the method and experimental descriptions and reorganized the main text and appendices. Revised and added passages are marked in blue in the PDF.

1. Clarifying the motivation and mechanism of Hit–Survival.

Section 3.2 distinguishes the observed-event support measured by Hit from the pre-trigger response history evaluated by Survival, explaining why candidate paths with similar endpoint support can differ in historical consistency. Section 3.4 clarifies how the interval objective distinguishes paths with similar predicted positions but different response histories. Appendix A.1 provides the normalization, reliability aggregation, and mathematical explanation of the bounded influence of clipped evidence.

2. Clarifying the ablation settings and table labels.

Section 4.4 and Table 4 separately evaluate Survival scoring, Hit, and the interval learning objective. Removing Survival scoring retains Hit, while removing the interval objective retains the scoring mechanism. We clarify the table labels to identify the component removed in each setting. Appendix C.3 explains how the variants are trained and specifies their data splits and initialization.

3. Explaining how candidate history affects correspondence probabilities.

We connect the tracking-error results in Table 4 with the candidate distributions in Figure 5 and bring key control settings and results from Appendix D into the main text. These controls fix network weights, candidates, and base scores while varying interval evidence. History shuffling preserves event locations, polarities, counts, and the multiset of interval durations, allowing the analysis to examine the role of temporal association.

4. Clarifying QART's inputs and the trajectory-head comparisons.

Section 3.3 explains the flow from refined correspondences and local motion features through temporal aggregation to 3D displacement prediction. It also explains viewing-direction ambiguity and the information provided by neighborhood geometry, with geometric analysis in Appendix F.1. Appendix F.2 details the GRU comparison using the same recurrent architecture and 2D frontend, and the Direct XYZ (direct head) comparison sharing QART's input features and temporal model. These comparisons examine the roles of candidate and neighborhood information and the choice of output representation.

5. Explaining dataset construction, screening, and evaluation.

Section 4.1 and Appendix C clarify EventTraj3D's native surface correspondences, trajectory construction, event-quality screening, query validity, and data splits. Algorithms C1 and C2 provide pseudocode for trajectory construction and clip selection. We also clarify the monocular inputs used for E-3DTrack, baseline retraining and initialization, and the pairing of 2D trackers with depth models. The evaluation description explains how 2D localization, absolute 3D position, and displacement errors are computed.

6. Expanding network architecture and computational-cost descriptions.

The appendices provide further details of the event and template encoders, candidate matcher, and QART, including module inputs and outputs, feature dimensions, feature fusion, and temporal aggregation. Appendix B.6 describes the latency measurement setup and reports accuracy and update latency for the full and lightweight configurations.

7. Improving notation, readability, and transitions.

We clarify mathematical symbols and the labels used in figures, strengthen transitions between method components, and connect the quantitative ablations with the candidate-history analysis. We also remove repeated explanations and reorganize the analyses accompanying Figures 5 and 6 into complete, coherent paragraphs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.