acceptodds
Withdrawn from ICLR 2027

StreamLRM: Test-Time Trained Memory for Streaming Multi-view 4D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Reconstruction Models (LRMs) have recently shown remarkable perfor- mance in feed-forward static 3D reconstruction. In this work, we explore how such powerful static models can be extended to volumetric video reconstruction, where synchronized cameras share the same multi-view geometry across time. A straightforward frame-wise extension treats each timestamp as an independent static scene, overlooking this temporal structure and repeatedly performing ex- pensive cross-view adaptation. We revisit TTT-based LRMs and reveal a key temporal asymmetry: scene content evolves with current observations, whereas fast-weight adaptation for cross-view interaction exhibits substantial temporal re- dundancy. Based on this observation, we propose StreamLRM, a simple training- free framework that temporally decouples TTT Update and Apply. We perform Update only at a reference timestamp and share the resulting fast weights across time, while each timestamp retains its own current-frame processing to capture evolving scene content. In this way, StreamLRM effectively extends pretrained static TTT-based LRMs to volumetric video without additional temporal modules or retraining. Surprisingly, this static-to-dynamic extension requires less computa- tion rather than more, achieving multi-fold inference acceleration with negligible reconstruction quality degradation on diverse multi-view video benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reconstruction, dynamic reconstruction, Large reconstruction model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ikPnOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper observes that fast weights in Test-Time Training (TTT) layers can be reused across timestamps in fixed-camera multi-view videos. To exploit this insight, the paper introduces StreamLRM, a training-free strategy that updates memory once and reuses it for subsequent frames. Compared with frame-wise inference using the static TTT-based baselines, StreamLRM achieves faster reconstruction and improved temporal consistency, with slightly lower rendering quality.

Critical strengths
  1. The observation of temporal redundancy in fast-weight adaptation is interesting and leads to a simple, effective method.
  2. The training-free strategy can be easily plugged into pretrained static TTT-based Large Reconstruction Models.

While I acknowledge these strengths, my initial score is primarily driven by the concerns below, particularly the unclear StreamLRM's positioning among existing dynamic scene reconstruction methods. I am willing to raise my score if the authors address my concerns during the rebuttal period.

Critical weaknesses
  1. The paper lacks sufficient analysis of the conditions under which the proposed reuse strategy succeeds or fails (see Q1 for more detail).
  2. While comparing with frame-wise 3D reconstruction baselines appropriately validates the reuse strategy, current experiments do not sufficiently establish StreamLRM’s position among existing dynamic scene reconstruction methods (see Q2 for more detail).
Questions for authors
  1. Could the authors further analyze when the reuse strategy becomes less effective?
    • Evaluating rendering quality and cosine similarity separately for static and dynamic regions could help identify where performance degradation occurs.
    • Under substantial appearance changes, such as fire-induced background color changes in the Flames scene of the Google Immersive dataset [1], would this lead to greater rendering quality degradation?
    • Would it become less effective with moving cameras, where less scene context may be reusable than in the fixed-camera setting?
    • Would it maintain stable reconstruction quality beyond 300 frames, such as for the dance scene used in Figure 7 (totaling 8,200 frames) or the bike scene (37,377 frames) of the SelfCap dataset?
      • Please provide the PSNR slope such as IGS's Figure 3 [2].
  2. Could the authors compare with existing dynamic scene reconstruction methods in terms of reconstruction speed, rendering quality, temporal consistency, and VRAM usage?
    • For example, valuable comparisons with 1) an online per-frame optimization method [3] that also aims to improve temporal consistency, and 2) NoPo4D discussed in the paper, would be valuable.
    • These experiments would clarify StreamLRM’s competitive standing among existing methods.

(Please carefully distinguish \citep and \citet in the paper.)

[1] Broxton et al. "Immersive light field video with a layered mesh representation." TOG 2020.

[2] Yan et al. "Instant Gaussian stream: Fast and generalizable streaming of dynamic scene reconstruction via Gaussian splatting." CVPR 2025.

[3] Yun et al. "Compensating spatiotemporally inconsistent observations for online dynamic 3D Gaussian splatting." SIGGRAPH 2025.

Reviewer_vHHXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends pretrained static TTT-based Large Reconstruction Models (TTT-LRM, TTT-LVSM) to streaming reconstruction of multi-view volumetric video captured by fixed, synchronized cameras. The authors observe that the fast weights produced by the TTT Update are largely reusable across timestamps: the evolving scene content is carried mainly by the current-frame queries, while the fast weights mostly provide shared cross-view context. Based on this, they propose StreamLRM, which runs the Update once at a reference frame, caches the fast weights, and reuses them for all later frames, so that only the current-frame processing and the Apply step are repeated. The method requires no retraining and adds no new modules. Experiments on four multi-view video datasets with two TTT backbones show roughly a 2× speedup over frame-wise reconstruction, with comparable reconstruction quality and better temporal consistency. Analysis experiments support the claim that the fast weights can be reused.

Critical strengths
  • Simple and practical. The method is training-free and plug-and-play, and it gives a consistent speedup with negligible quality loss across backbones and datasets. Because the cost of volumetric video grows with sequence length, a trick that costs nothing to adopt has direct practical value.

  • An interesting empirical finding. The observation that the TTT Update is essential, but repeating it at every frame is largely redundant, helps clarify what queries and fast weights each do in TTT-based reconstruction, and it may inform future 4D designs.

Critical weaknesses

[Contribution and positioning]

  • The contribution is a single inference-time caching trick: reusing the frame-0 fast weights across frames. There is no new module, no new training objective, and no theoretical analysis; the novelty rests on one empirical observation.
  • The speedup is bounded by 1 + C_u (update cost) / C_b (other cost). In this setting the bound is about 2×, so "multi-fold acceleration" in the abstract is overstated.
  • The reason W can be reused is explained only at the level of observed behavior, without any mechanistic analysis or verification.
  • The outputs are still Gaussians predicted independently for each frame, with no correspondence across time. The "4D" result is effectively a sequence of 3D reconstructions, which limits downstream uses such as editing or compression.
  • There is no mechanism for deciding when W should be refreshed. The problem of large scene changes, which the authors acknowledge in the limitations, is left unaddressed.

[Evaluation design]

  • With W_0 frozen, cross-view fusion at frame t relies on frame 0, so the multi-view correspondences of the dynamic foreground are never re-established. Yet no metrics are reported for foreground or dynamic regions, and full-image metrics are dominated by the static background.
  • The central evidence, cosine similarity above 98.5%, has no reference point, such as the similarity obtained with a cross-scene W. The cross-scene transfer in Fig. 4 is shown only qualitatively.
  • There are no stress tests. Objects entering the scene, large appearance or lighting changes, and large motions are not covered.
  • The reference frame is always frame 0. Sensitivity to this choice is not studied, for example when the foreground occludes part of the background in the reference frame.
  • Only one camera (camera 0) is held out per scene, there are 15 sequences in total, and no variance or significance is reported.
  • Only six input views are tested, whereas tttLRM was trained with 16–64 views. The effect of the number of views is not explored.

[Baselines and comparisons]

  • There are no naive acceleration baselines at equal latency (about 520 ms), such as lower resolution, fewer layers, or reconstructing keyframes and interpolating. It is therefore unclear whether reusing W actually beats simple alternatives.
  • The feed-forward 4D methods discussed in related work (Nopo4D, SplatVoxel) are not compared against. No per-scene-optimization 4D method (e.g., 4DGS, SpacetimeGS) is reported as a quality reference either.
  • WorldMirror and DepthSplat are evaluated at lower resolution, so they are not directly comparable, yet the paper describes this as a "fair comparison." DepthSplat's 10.79 dB PSNR on ENeRF is also strange, was it misconfigured?
Questions for authors
  • Can you report PSNR, SSIM, and LPIPS separately for dynamic and static regions (for example, using foreground or motion masks), compared with baseline methods?
  • Can you also split tOF into dynamic and static regions, to show where the gain in temporal consistency comes from?
  • Can you report the cosine similarity between Apply(Q_t, W_t) and Apply with a cross-scene W, and with W_init, as well as the PSNR change when a cross-scene W is used? This determines whether the reported 98.5% similarity is actually discriminative.
  • The paper attributes reusability to W mainly encoding camera geometry. However, in Fig. 4, it seems transferring W between scenes captured by the same camera rig leaves visible source-scene remnants?
  • What quality does TTT-LRM reach when tuned to about 520 ms per frame via lower resolution, fewer layers, or keyframe reconstruction plus interpolation? Is StreamLRM clearly better at equal latency?
  • Can you compare with Nopo4D and SplatVoxel, and report at least one per-scene-optimization 4D method as a quality reference?
  • On sequences where objects enter or leave, or where appearance or lighting changes noticeably, does quality degrade over time?
  • How do results change if the reference frame is taken from the middle of the sequence, or is a frame with significant foreground occlusion?
  • Why can WorldMirror and DepthSplat only be run at low resolution? Does DepthSplat's 10.79 dB PSNR on ENeRF indicate a failed run?
Reviewer_y1rbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamLRM accelerates streaming multi-view reconstruction by computing TTT fast weights at the first timestamp and reusing them for subsequent frames. Current-frame features are still processed, allowing scene content to change without repeatedly updating the memory. The method requires no additional training and is evaluated on four datasets. This multi-view setting is useful.

Critical strengths

The method is simple and easy to integrate into existing TTT-based models. It reduces latency from 1091.4 to 519.8 ms/frame while changing average PSNR from 26.02 to 25.96 dB. This setting is absolutely meaningful and timely.

Critical weaknesses
    1. The significance of the efficiency improvement is not sufficiently established. Halving computation while preserving quality is useful, but the demonstrated contribution is primarily an inference optimization for existing TTT-based models. Processing still takes approximately 520 ms per timestamp, which does not support real-time reconstruction of typical video streams.
    1. It is a straightforward way to inference with multiview images, but multiview images does not contribute much to the inference at all. Multi-view images are not burden and they should bring substantial improvements compare to the baselines.
Questions for authors
    1. How does the quality–runtime trade-off compare with alternative reconstruction methods at matched resolutions? (baselines have different resolutions)
    1. Can you report reconstruction and temporal errors separately for dynamic foregrounds and static backgrounds, including sequences where reference-memory reuse fails?
    1. Multiview is just a setting. Have the authors consider to use the information extended by multiview?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.