acceptodds
Withdrawn from ICLR 2027

Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent years, 3D visual foundation models, pioneered by pointmap-based approaches such as DUSt3R, have attracted a lot of interest, achieving impressive accuracy and strong generalization across diverse scenes. However, these methods are inherently limited to recovering scene geometry only at the discrete time instants when images are captured, leaving the scene evolution during the blind time between consecutive frames largely unexplored. We introduce Interp3R, to the best of our knowledge, the first method that enhances pointmap-based models to estimate depth and camera poses at arbitrary time instants. It leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous geometric representations. Depth and camera poses are then jointly recovered by aligning the interpolated pointmaps together with those predicted by the underlying frame-based models into a consistent spatial framework. We train Interp3R exclusively on a synthetic dataset, yet demonstrate strong generalization across six datasets, both synthetic and real. Compared with the best two-stage baseline, Interp3R reduces absolute relative depth error by 15%–32% on DSEC and absolute trajectory error by up to 51% on EDS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event Camera, 3D Geometry Estimation, Continuous-time Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_icMBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to predict a point map at an interpolated timestamp between two input views. To this end, it leverages an event stream to capture the motion between the two views, and introduces several simple modules that enable point map prediction at the intermediate timestamp.

Critical strengths
  • While existing methods learn using events for a specific timestamp (usually the timestamp corresponding to the input view), this paper makes good use of the characteristics of events to predict the point map of the intermediate interpolated time well.
  • The experimental results are well presented, and it outperforms existing methods on various datasets, recording the highest numbers.
Critical weaknesses
  1. Limited contribution. In my opinion, the main weakness is that the contribution is quite limited. The event representation follows existing encoding schemes, and the core idea appears to be simply taking the events corresponding to a target time as a condition and feeding them into the Align3R decoder to predict the point map. Although a few simple modules are proposed for this purpose, they are not specifically designed around the characteristics of event data, and the method appears to be mainly a combination of two existing modules that happens to work well.

  2. Outdated related work. The related work section is largely outdated. For dynamic scenes, recent works such as D²USt3R, POMATO, DPM, and St4RTrack [A,B,C,D] predict point maps aligned to a specific timestamp, and N-view methods such as VGGT and Depth Anything 3 [E,F] also exist. These works also follow the approach of MonST3R to handle dynamic regions. More recently, models such as Trace Anything and D4RT (Open D4RT) [G,H] predict point maps at interpolated timestamps on top of models like VGGT. A comparison with these models would strengthen the paper.

  3. Missing comparison with Dynamic DUSt3R. The paper mentions Dynamic DUSt3R but does not compare against it. Since Dynamic DUSt3R predicts an interpolated point map exactly at the middle timestamp, results at the middle timestamp should also be reported to show how the proposed method compares.

  4. Unclear motivation for intermediate timestamps. While I understand the goal of predicting point maps at intermediate timestamps, it would be helpful to explain why this is useful and in which applications. When I think about how such intermediate predictions would actually be used, no clear use case comes to mind. I suspect this may also be why existing methods either predict at the timestamp of a specific view, or align to a specific view to obtain outputs such as point tracks, rather than predicting point maps at intermediate timestamps.

  5. Missing details on the event voxel. The paper does not clearly explain how the event voxel is constructed. It states that it follows existing EVFI methods, but it should clarify whether the voxel has the shape (x, y, z) or (x, y, t). If it is (x, y, t), the main paper should explain how the event voxel is combined with point map prediction. I consider this the most important part of the method, yet the concrete details are omitted.

  6. Questionable plug-and-play claim. When comparing MonST3R + Interp3R with the w/o event setting in Tab 3, the method performs worse on most depth metrics, which is disappointing. If the paper claims plug-and-play capability (as in Figure 1), the method should work in general, so this result is concerning. It is also hard to attribute this to a distribution gap, since the module only takes point maps as input. Therefore, I am not sure the plug-and-play claim is fully supported.

[A] Han, Jisang, et al. "D^ 2USt3R: Enhancing 3D Reconstruction with 4D Pointmaps for Dynamic Scenes." arXiv e-prints (2025): arXiv-2504.

[B] Zhang, Songyan, et al. "Pomato: Marrying pointmap matching with temporal motions for dynamic 3d reconstruction." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[C] Sucar, Edgar, et al. "Dynamic point maps: A versatile representation for dynamic 3d reconstruction." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[D] Feng, Haiwen, et al. "St4rtrack: Simultaneous 4d reconstruction and tracking in the world." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[E] Wang, Jianyuan, et al. "Vggt: Visual geometry grounded transformer." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[F] Lin, Haotong, et al. "Depth anything 3: Recovering the visual space from any views." arXiv preprint arXiv:2511.10647 (2025).

[G] Liu, Xinhang, et al. "Trace anything: Representing any video in 4d via trajectory fields." International Conference on Learning Representations. Vol. 2026. 2026.

[H] Zhang, Chuhan, et al. "Efficiently reconstructing dynamic scenes one d4rt at a time." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

See weakness

Reviewer_Bd8aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Interp3R, which extends pointmap-based 3D foundation models (DUSt3R / MonST3R / Align3R) to continuous-time geometry estimation. Given two frames and the asynchronous event stream between them, Interp3R freezes the pointmap backbone and interpolates pointmaps at an arbitrary query time τ from two directions (0→τ and 1→τ), using explicit time encoding and zero-convolution fusion of event, pointmap, and RGB features. Depth and camera poses are then jointly recovered by coarse-to-fine global alignment (MonST3R objective with RAFT flow in the coarse stage; E-RAFT flow, interpolated nodes, and a translation prior in the fine stage), fusing four symmetrized pointmap measurements by pixel-wise confidence. Training uses only synthetic PointOdyssey triplets (spans 2–10 frames). Evaluation spans six datasets — depth on PointOdyssey / Sintel / Bonn / DSEC (real events), pose on Sintel / Bonn / ECD / EDS (real events) — with skips 1/3/7/15, against two-stage baselines (RIFE / TimeLens / VDM-EVFI + Align3R). Reported gains: 15%–32% Abs Rel reduction on DSEC and up to 51% ATE reduction on EDS.

Critical strengths
  1. Novel and practical combination: within the pointmap feed-forward family, this is the first approach to use asynchronous events for bidirectional pointmap interpolation and joint estimation of depth and camera poses at arbitrary time instants. The design — explicit time encoding, frozen-backbone plug-and-play, symmetric four-pointmap confidence fusion, coarse-to-fine alignment, compatibility with MonST3R/Align3R — is clean and reusable, with genuine engineering value.
  2. The experimental story is solid where presented: synthetic-only training generalizes to real-event data (DSEC depth; ECD/EDS pose), with consistent wins over two-stage baselines at every setting; the ablation isolates the event contribution (Abs Rel reduction grows from 4% to 15% as skip increases; −55% ATE at skip=7). I independently verified all tables — internally consistent.
Critical weaknesses
  1. The headline claim ("arbitrary / continuous-time queries") lacks quantitative evidence in the main text: all quantitative results are aggregates over the skipped frames, with no τ-disaggregated errors or error-vs-τ curves; the only arbitrary-time evidence (supplementary Figure 7) is qualitative and not included in the provided PDF. If the evaluation is in fact midpoint-only (τ=0.5), the core claim is effectively untested — this is the most critical evidence gap.
  2. Positioning is incomplete and the "first" claims are overstated: the closest competitor EAG3R (NeurIPS 2025; MonST3R + events, depth and pose) is never compared; recent 4D pointmap works (D²USt3R, Dynamic Point Maps, St4RTrack, UFO-4D, Flow4R), event-based continuous reconstruction (E-NeRF, Dynamic EventNeRF, E-NeMF, Event3DGS, E-3DGS), and continuous-time event trajectory estimation (Mueggler et al.; Sommer et al.) are absent from the related work, while "for the first time" appears four times without scoping.
Questions for authors

1.The core selling point—handling "arbitrary" timestamps—lacks quantitative evidence. The "skip" mechanism alters the temporal span rather than the specific query timestamp τ, yet the set of τ values ​​used for evaluation is not specified.

2.There is no comparison with the closest competitor, EAG3R (which combines MonST3R with event data, depth, and pose), nor with other event-based depth estimation methods such as EventVGGT.

3.The ablation study is limited to only three lines; it fails to analyze key components such as temporal encoding, the coarse-to-fine two-stage process, confidence fusion, bidirectionality, and the distribution of training spans.

4.The paper lacks a discussion on limitations or failure cases, making the concluding claim of "thorough experiments" an overstatement.

5.Implementation details are insufficient: the paper omits information regarding voxelization resolution (or bin counts), encoder/decoder dimensions, the specific form of temporal encoding, the values ​​used for α and w_trans, and inference latency.

Reviewer_qbVuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Interp3R, which extends frame-based pointmap reconstruction to depth and camera-pose estimation at arbitrary intermediate timestamps using events. The interpolation network combines RGB features, source pointmaps, confidence maps, and event features through zero-convolution conditioning, with an explicit target-time encoding in the prediction heads. The resulting pointmaps are incorporated into a coarse-to-fine global alignment procedure, which includes a scale-invariant prior to preserve the relative translational structure of the coarse camera trajectory.

The paper addresses an interesting problem and reports encouraging improvements over the evaluated two-stage baselines. Although the individual architectural components are largely established, their integration provides a useful extension of existing pointmap-based reconstruction methods.

Critical strengths
  1. An interesting use of events for arbitrary-time geometry estimation. Extending reconstruction beyond RGB frame timestamps is a useful direction. Events provide observations within the temporal gaps between frames, making them a natural source of information for estimating intermediate geometry.

  2. Encouraging generalization beyond the training domain. The interpolation model is trained on PointOdyssey and evaluated on other datasets, including datasets with real event measurements, without dataset-specific network fine-tuning. The inference-time replacement of Align3R with MonST3R also provides evidence of compatibility with another pointmap backbone.

  3. Evaluation across multiple benchmarks. The paper evaluates both depth and camera-pose estimation across synthetic and real datasets and multiple frame skips. The results generally compare favorably with the evaluated interpolation-plus-reconstruction pipelines. The event-input ablation also supports the usefulness of events, particularly as the frame gap increases.

  4. Clear overall presentation. The motivation and main pipeline are easy to follow. The supplementary material provides useful additions, including the translation-prior formulation, runtime measurements, and further qualitative results.

Critical weaknesses
  • The method design is not sufficiently explained or experimentally justified. Although the overall pipeline is understandable, the description of the additional conditioning mechanisms is incomplete, and the experiments do not adequately isolate the contributions of the proposed components. Consequently, it remains difficult to assess why the particular combination of inputs, temporal conditioning, and alignment constraints is necessary, and which components are responsible for the reported gains. This limits my confidence in the design rationale and the attribution of the improvements from event camera.

  • The evaluation does not sufficiently characterize the claimed continuous-time capabilities under challenging dynamics. The results demonstrate improvements at sampled intermediate timestamps, and the supplementary material provides qualitative examples of non-uniform queries and dynamic scenes. However, the evidence remains limited regarding temporal accuracy across arbitrary query positions and robustness under substantial or non-uniform motion. The physical time spans are also not consistently reported across training and evaluation. As a result, the operating range and limitations of the proposed continuous-time reconstruction framework are not yet clearly established.

Questions for authors

Questions

  1. The supplement gives the zero-convolution update, but the exact fusion scheme is still unclear. How are the pointmap, confidence, and event features combined, and which decoder layers receive them? Please also specify how the time embedding is constructed and injected into H, and how many temporal bins are used for events.

  2. Table 3 examines the event input and backbone choice, but does not show the individual contributions of the other additions. Could you provide separate ablations for source-pointmap conditioning, source-confidence conditioning, explicit time conditioning, and L_trans, with otherwise matched settings? When removing the time embedding, event truncation at the query time should be retained. For L_trans, please also plot the coarse trajectory and the refined trajectories with and without the prior, for example on the Figure 13 case at skip = 15.

  3. The DSEC sampling rate is specified, but how long are the 2–10-frame training triplets in seconds? What actual time intervals do the skip settings correspond to on the other test datasets? Please provide a table with the effective sampling rates and endpoint-gap ranges, including any temporal subsampling and the time step used for event simulation.

  4. Figure 7 shows predictions at non-uniform query times, but their accuracy is not reported. For a fixed collection of source-frame pairs, how do depth and pose errors vary with the relative query time τ? Please report errors across query positions with available ground truth, including positions near the endpoints and the middle of the interval. Non-uniform queries should also be included where ground truth is available.

  5. Do the improvements hold on independently moving objects? Whole-image depth scores do not show how well these regions are reconstructed. On an existing test set with suitable annotations, please report errors separately for the static background and moving foreground, and break down the foreground results by motion magnitude.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.