acceptodds
Withdrawn from ICLR 2027

T-LRM: Temporal-aware LRM for multi-view dynamic scene reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Reconstruction Models (LRMs) have recently emerged as the dominant paradigm for feed-forward 3D reconstruction. However, existing 3D foundation models are primarily designed for static scenes or monocular dynamic inputs, making them difficult to extend to high-quality multi-view dynamic scene reconstruction. A straightforward frame-by-frame inference strategy is computationally expensive and fails to exploit the strong temporal redundancy inherent in multi-view videos. To address these limitations, we propose T-LRM, a Temporal-aware LRM framework that extends existing 3D foundation models to temporally aware multi-view dynamic reconstruction in a training-free manner. Our key insight is that multi-view dynamic videos contain substantial spatial redundancy across time. By identifying dynamic regions while sharing temporally consistent static regions, redundant computation can be largely eliminated and cross-temporal interference can be effectively suppressed. Based on this observation, we introduce a Temporal-aware Token Merging strategy that anchors the scene geometry using shared static tokens while reconstructing dynamic objects with timestamp-specific dynamic tokens. The proposed framework is lightweight, plug-and-play, and can be seamlessly integrated into different feed-forward reconstruction architectures without retraining. Extensive experiments on multiple benchmark datasets demonstrate that T-LRM achieves multi-fold inference acceleration while maintaining comparable reconstruction quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision, LRM, Dynamic scene reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8scAOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to accelerate feed-forward dynamic scene reconstruction from a fixed, synchronized, and calibrated multi-camera system. The authors propose T-LRM, a training-free framework that first classifies image patches as static or dynamic based on their appearance variation relative to the temporal mean. Before global interaction, static tokens at corresponding locations are averaged across timestamps and processed jointly, while dynamic tokens are retained separately for each timestamp. The shared static group and the timestamp-specific dynamic groups independently undergo global attention or TTT fast-weight interaction, after which replication and reordering restore the original token layout. The method is evaluated on N3DV, MeetRoom, ZJU-SelfCap, and ENeRF, and is applied to TTT-LRM and WorldMirror. The reported results show approximately twofold inference acceleration with a small average loss in reconstruction quality and a reduction in temporal optical-flow error.

Critical strengths
  1. The paper addresses a practical problem with a simple, training-free method. Repeatedly processing static backgrounds independently at every timestamp introduces substantial redundancy in videos captured by fixed cameras. Exploiting this structure to reduce global interaction computation is sensible. The method requires neither additional training data nor a dedicated temporal modeling network, making it relatively straightforward to adopt.

  2. The experiments include a useful control for ordinary frame parallelism. Table 2 shows that ordinary frame parallelism provides only approximately 1.11-1.12x overall acceleration, whereas the proposed method further reduces the runtime of Global Token Interaction substantially. This is more informative than simply comparing frame-wise and chunk-wise inference, and supports the conclusion that the gains do not arise entirely from batching.

  3. Results across two global interaction mechanisms provide initial evidence of general applicability. Table 3 demonstrates acceleration on N3DV for both WorldMirror and TTT-LRM, supporting applicability to attention-based and TTT-based backbones. The appendix also clarifies that the main experiments use a shared classification threshold across scenes, which strengthens the experimental setup.

Critical weaknesses
  1. The methodological novelty appears limited and primarily concerns incremental changes to the inference computation. In my view, the efficiency gains mainly arise from two design choices: organizing multiple timestamps into temporal chunks and reusing computation for static tokens across time, and processing static and dynamic tokens in separate global interaction groups to reduce computation. These choices have practical value, but the overall contribution appears closer to an optimization of the computation performed by existing 3D reconstruction backbones. Table 2 already establishes that the gains are not solely due to ordinary frame parallelism. The authors should therefore more clearly establish what static temporal reuse and grouped interaction contribute beyond existing token merging and computation reuse strategies.

  2. The positioning of the method and the motivation for its temporal design need further clarification. The method accelerates reconstruction at individual timestamps by sharing static tokens across time, ultimately producing a corresponding set of 3D Gaussians for each timestamp. It does not construct a unified, continuous-time 4D Gaussian representation or explicitly model Gaussian trajectories or deformations across frames. This is a reasonable design for accelerating feed-forward reconstruction, but the rationale for this particular temporal processing strategy remains insufficiently explained.

  3. The ablations do not isolate the effects of temporal reuse and the removal of static-dynamic interaction. According to Section 3.3 and Appendix B, the method simultaneously introduces two changes: sharing static tokens across timestamps and removing Global Token Interaction between the static and dynamic groups. For attention-based backbones, the interaction cost changes from T(S+D)^2 to S^2+TD^2, so savings arise from both static computation reuse and the removal of cross-group interaction terms. Table 2 rules out an explanation based solely on frame parallelism, but does not further disentangle these two changes. The experiments therefore support the effectiveness of the complete computation strategy, while leaving the incremental benefit of temporal-aware merging over simple grouped interaction unclear.

  4. Full-image average metrics do not sufficiently establish preservation of dynamic details and temporal consistency across chunk boundaries. The method is motivated by the high proportion of static background in the input videos. However, this also means that full-image PSNR, SSIM, and LPIPS can be dominated by background regions. Noticeable degradation on moving subjects, occlusion boundaries, or fine structures may have only a limited effect on these aggregate metrics.

  5. The timing scope is unclear, and throughput, output latency, and memory costs should be reported together. The Overall runtime in Table 2 exactly equals the sum of Frame-Attention and Global Token Interaction; for example, the 20-frame configuration reports 309+197=506 ms/frame. The manuscript does not clarify whether this includes patch classification, token merging and restoration, other network modules, Gaussian decoding, and rendering. It is therefore difficult to determine whether these numbers represent the runtime of the core modules or the complete reconstruction pipeline.

Questions for authors
  1. Methodological novelty: What are the key contributions of static token reuse within temporal chunks and separate static-dynamic global interaction beyond existing token merging and computation reuse strategies? Could representative, applicable alternatives be compared using the same backbone, inputs, and evaluation protocol, reporting reconstruction quality, runtime, and memory consumption to establish the incremental value of the proposed method?

  2. Method positioning and design motivation: The method shares static features across time while reconstructing 3D Gaussians separately for each timestamp. Could the authors further explain the rationale for this temporal processing strategy and how it balances motion fidelity and temporal consistency without explicitly modeling Gaussian trajectories or deformations across frames? Please clarify its advantages and limitations in computational efficiency, dynamic reconstruction quality, and applicable scenarios.

  3. Attribution of the design contributions: Could controlled experiments separately quantify the effects of temporal static-token reuse and the removal of static-dynamic global interaction on speed and quality? For example, how does the method perform with the same static-dynamic partition but without temporal reuse? How does a sharing scheme that retains information exchange between static and dynamic tokens perform? These comparisons would help establish the individual contributions of the two choices and the need to combine them.

  4. Dynamic regions and temporal consistency: Beyond full-image averages, could the authors report reconstruction quality on moving foregrounds, motion boundaries, and occlusion regions, together with separate temporal errors within chunks and across chunk boundaries? Can these results establish whether improvements in static backgrounds mask degradation on moving subjects? Please also specify the optical-flow estimator, error units, and frame-pair selection protocol used for tOF.

  5. Complete efficiency assessment: Does the Overall runtime in Table 2 include patch classification, token merging and restoration, other network modules, Gaussian decoding, and rendering? Please define the timing boundaries and report complete pipeline runtime, processing time per temporal chunk, and latency to the first output. Memory consumption and gains over ordinary frame parallelism under a fixed memory budget would also help characterize the efficiency-resource trade-off of chunk-wise inference.

Reviewer_SJbAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

T-LRM is a training-free framework for accelerating feed-forward multi-view dynamic scene reconstruction from synchronized, calibrated, fixed-camera videos. It classifies spatially aligned patches as static or dynamic based on temporal appearance variation. Within each Global Token Interaction block, static-token features are averaged across a temporal chunk and processed once, while dynamic tokens are processed independently for each timestamp. The outputs are then unmerged to restore the original token layout. Frame-Attention is retained, whereas global interaction between static and dynamic groups is omitted. The framework is instantiated on both the TTT-based TTT-LRM and the attention-based WorldMirror backbone without retraining. Experiments across four datasets, with additional WorldMirror validation on N3DV, report roughly twofold inference speedups, small reductions in reconstruction quality, and improved aggregate temporal optical-flow error.

Critical strengths
  1. Practical acceleration without retraining. Table 2 reports a 2.14x speedup with only a 0.11 dB PSNR decrease at 20 frames, substantially exceeding naive frame parallelism.
  2. Evidence across datasets and backbones. Experiments evaluate TTT-LRM on four datasets and additionally validate the method on the attention-based WorldMirror backbone on N3DV, providing initial evidence of applicability across architectures.
Critical weaknesses

1.The added value over prior temporal token reuse and merging is insufficiently established. Eventful Transformers (arXiv:2308.13494) selectively update changing tokens, while VidToMe (arXiv:2312.10656) merges and restores cross-frame tokens using destination-token values rather than averaging. These works differ in task and mechanism, particularly from T-LRM's TTT implementation. A methodological comparison and a controlled representative-frame reuse baseline would clarify the added value of the proposed design. 2. The sources of acceleration are not fully disentangled. The method both reuses static computation and removes static-dynamic exchange within Global Token Interaction, while retaining Frame-Attention. Processing the same static and dynamic groups independently at each timestamp, without temporal averaging or copying, would help disentangle the effects of grouping and sharing on speed and quality. An optional dynamic-to-static attention comparison could assess the value of restoring limited context. 3. The sources of acceleration are not fully disentangled. The method both reuses static computation and removes static-dynamic exchange within Global Token Interaction, while retaining Frame-Attention. Processing the same static and dynamic groups independently at each timestamp, without temporal averaging or copying, would help disentangle the effects of grouping and sharing on speed and quality. An optional dynamic-to-static attention comparison could assess the value of restoring limited context.

Questions for authors
  1. How does T-LRM differ methodologically from Eventful Transformers and VidToMe, and what does temporal averaging add over representative-frame reuse with the backbone and masks fixed?
  2. Can you compare the original backbone, a variant that processes static and dynamic groups separately at every timestamp without temporal sharing, and the full method, using the same static/dynamic masks and reporting both reconstruction quality and runtime?
  3. What representation is used for patch classification, and how do static ratios, runtime, and local or cross-chunk quality vary across scenes?
  4. Which stages are included in the reported overall runtime, and how does amortized per-frame processing time relate to total chunk processing time?
Reviewer_dRc8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method to significantly speed up the 3D+time reconstruction of multi-view videos. The method both increases the speed and reduces the memory footprint of existing models (so no retraining), in compatible situations. The situations are a little narrow: the cameras need to be static, and the paper + validation are focused on situations where there are multiple cameras.

The crux of the method is temporal token reuse, where 2D change-detection classifies which pixels didn't change over the duration of the sequence, then allowing the corresponding tokens to be handled separately from the default ("dynamic") tokens. Static ones are passed through the global interaction block (attention or the TTT update). Dynamic ones go through the block separately for each timestamp, and they only attend to each other at that stage. Later, they rebuild the original layout (“unmerging”) before the decoder.

The main finding is that a pretrained LRM tolerates having its global block split this way without retraining.

Critical strengths

I have little doubt that this approach will work well to save computations and memory for compatible situations (static cameras). Static cameras come up quite often, e.g. CCTV networks.

I also think the proposed method would work on monocular video settings, again as long as the camera is static. This was not demonstrated, but the proposed token re-use doesn't leverage or need the multiple cameras, as far as I can see.

The approach works with both attention-based and TTT-based backbones.

Last, for the most part, the method degrades gracefully, as less of the scene is static and amenable to token re-use. It just doesn't save as much in those situations.

Critical weaknesses

While the approach shows a good understanding of where computations are potentially being wasted in the global interaction block, it does not surprise me that static backgrounds can be handled differently from the moving parts of the scene to save processing.

Background subtraction for static cameras is obviously used for introductory computer vision courses. Such a naive approach should do badly compared to the proposed method. But this approach isn't doing that much more, despite having access to multiple views. I was able to find very similar token-reuse of background pixels for static-cameras in

  • Eventful Transformers (Dutson ICCV 2023)
  • Don't Look Twice (Rohan Choudhury Neurips 2024)

Is there a meaningful difference here?

No step here compares views, enforces agreement between them, or reasons in 3D, which could have set this apart.

Related, the assumption that there are multiple cameras and that they are all fixed limits the scope somewhat.

Questions for authors

Why not leverage multiple camera's views of the same volume to reuse tokens based on 3D space, like voxels or gaussian-splat primitives?

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

Thank you for your thorough review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.