acceptodds
Withdrawn from ICLR 2027

ReStream: Fast Streaming Reconstruction with Incremental Dynamic Token Updates

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Volumetric video reconstruction aims to recover dynamic 3D scenes from synchronized multi-view videos, which are commonly captured by fixed camera rigs. Recent Large Reconstruction Models (LRMs) offer a promising feed-forward solution, yet applying them frame by frame incurs substantial redundant computation, as large portions of the scene remain unchanged throughout the video. We introduce \ourname, a training-free framework that converts frame-wise LRM inference into efficient streaming reconstruction by exploiting this temporal redundancy. Our key observation is that, under fixed-view capture, representations of static scene content can be effectively reused across timestamps, while only dynamic regions require continuous updates. Accordingly, \ourname decouples temporal computation into static reuse and dynamic token updates: static representations are carried forward from the reference frame, whereas dynamic tokens are selectively recomputed from current observations. This principle is independent of a specific interaction mechanism and can be readily applied to both attention- and TTT-based reconstruction models without additional temporal modules or retraining. Extensive experiments on diverse multi-view dynamic benchmarks demonstrate substantial inference acceleration with minimal degradation in reconstruction quality, establishing \ourname as a simple and effective framework for streaming feed-forward volumetric video reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Foundation model, 4D scene reconstruction, Large reconstruction model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KEozOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReStream, a training-free approach to accelerating volumetric video reconstruction from synchronized, fixed-view cameras. It caches representations from a reference frame, identifies changed patches using embedding differences, and updates dynamic tokens together with a small subset of static anchors. The approach supports both attention-based and test-time-training-based reconstruction backbones.

The reported results demonstrate a promising efficiency–quality trade-off: approximately 3.6× acceleration over TTT-LRM, with mean PSNR decreasing from 26.02 to 25.91 dB and temporal optical-flow error decreasing from 0.282 to 0.192. The main contribution is a practical adaptation of temporal computation reuse to multi-view reconstruction. The principal questions concern its contribution beyond established token-reuse methods and whether the evaluation sufficiently establishes preservation of dynamic content.

Critical strengths
  • Technical novelty and innovation: The combination of reference-frame caching, selective dynamic updates, and static anchors is simple and technically plausible. Applying the strategy to both attention and TTT interaction mechanisms is a useful contribution, particularly because it requires no additional model training.

  • Experimental rigor and validation: The matched-backbone comparisons are the strongest evidence. Appendix A.4 specifies identical pretrained weights, input views, preprocessing, and evaluation frames. Experiments cover four datasets, and the paper includes ablations on the change threshold, reference-refresh interval, and anchor ratio. Table 4 explicitly includes initialization and refresh costs.

  • Clarity of presentation: The overall pipeline is easy to understand, and Figure 3 communicates the computation-reuse mechanism effectively. Appendix A.1 usefully distinguishes frame attention, global attention, and TTT updates.

  • Significance of contributions: A roughly 3.6× speedup with a small average reconstruction-quality reduction would be valuable to practitioners. The additional results on WorldMirror suggest that the approach is not restricted to one interaction architecture.

Critical weaknesses

W1. The methodological contribution relative to existing temporal token reuse is insufficiently established.

Eventful Transformers, ICCV 2023 already identifies changed tokens, reuses previous computations, and selectively updates transformer operations, generally without retraining. This does not establish equivalence to ReStream: multi-view geometry, static anchoring, and TTT adaptation may introduce meaningful differences. However, the paper does not explain these differences sufficiently to establish what new methodological insight enables its results.

The patch-level decomposability experiment does not fully resolve this concern. Independently processing complementary subsets of the same frame differs from combining current dynamic representations with cached representations computed under an earlier scene configuration. Moreover, spatially distributed random subsets can preserve scene-wide context, whereas a dynamic mask may be concentrated around a moving object. Figure 2 therefore supports a narrower observation than the general justification presented in the text.

W2. The temporal evaluation does not sufficiently separate background stabilization from preservation of dynamic content.

The reduction in tOF is encouraging, but both tOF and the reconstruction metrics are averaged over the image. Since the method explicitly reuses a large static region, improvements there could outweigh degradation around moving objects or occlusion boundaries.

This does not make tOF inappropriate: it compares reconstructed motion against ground-truth motion. The concern is that its aggregate value cannot establish where the improvement occurs. A breakdown of existing predictions into independently defined static and dynamic regions would directly test the paper’s claim that selective reconstruction preserves dynamic scene changes.

Figure 6 also needs a precise definition of its residual maps. If these show unaligned image differences, lower residuals alone do not establish better temporal fidelity, because genuine motion should produce nonzero differences.

W3. Several implementation ambiguities prevent a confident interpretation of the selective-update algorithm.

The paper does not sufficiently specify which intermediate representations are cached at each layer or how those caches interact with normalization, residual connections, and changing masks. For TTT, Equation (9) uses WinitW_{\mathrm{init}}, but its relationship to the learned initialization and reference-frame fast weights needs clarification.

Questions for authors
  1. What distinguishes ReStream from applying established temporal token gating and caching to a reconstruction backbone? Does the decomposability observation hold for the spatially concentrated masks actually produced by your method?

  2. Can you report reconstruction and temporal errors separately for static and dynamic regions on representative existing evaluation sequences, using an independent mask? What precisely do the residual maps in Figure 6 measure?

  3. What is �0��0� in Equation (9), and which tensors are cached at each layer? Is the 100% anchor row in Table 6 an actual selective-update run or the original full-inference baseline?

  4. Is camera 0 always excluded from the input views? What evaluation resolution and optical-flow estimator are used across methods, and which pipeline components are included in the main latency measurements?

5 comments

We thank the reviewer KEoz for recognizing ReStream’s practical value, training-free applicability across attention and TTT backbones, matched-backbone evaluations, and clear presentation. Below, we address your concerns with additional evidence and clarifications.

W1. Relationship to Eventful Transformers and token decomposability.

(1) Relationship to Eventful Transformers.

We agree that token gating, computation reuse, and training-free selective updates are established by Eventful Transformers. Our methodological insight is to separate the static context needed for dynamic reconstruction from the static representations that need updating. Sparse reference static anchors provide scene context for dynamic reconstruction, while final static representations remain unchanged.

Eventful Transformers reduces repeated attention computation through gating, buffering, and incremental updates. ReStream instead excludes most static tokens from current-frame cross-view interaction, retaining sparse anchors to support geometric consistency with the cached static scene. This distinction is summarized below.

AspectEventful TransformersReStream
Global interactionIncremental updates to cached attention computationsRestricts cross-view interaction to dynamic tokens and sparse static anchors
Reference statePer-token state from its most recent updateReference-frame features within each refresh interval
Context handlingBuffered token and attention statesFull static K/V within views; sparse reference anchors across views
Context versus retained updatesGates and buffers selectively update intermediate statesAnchors participate as context, but their updated outputs do not replace final static representations
Demonstrated mechanismsStandard attentionAttention and TTT, including selective fast-weight Update

For TTT, restricting the tokens participating in Update reduces fast-weight adaptation work itself; merely reducing queries in Apply would not achieve this. Together, these designs specify how to preserve reconstruction context while reducing global computation, beyond choosing which tokens to recompute.

Table 6 directly supports the role of sparse reference context: adding 10% static anchors improves PSNR by 0.64 dB, reaching 29.12 dB compared with the full model’s 29.23 dB, at only 0.01 s/frame additional cost.

Anchor RatioPSNR ↑SSIM ↑LPIPS ↓Time (s/frame) ↓
0%28.480.8820.1670.29
5%28.960.8920.1600.30
10%29.120.8950.1580.30
20%29.170.8950.1580.35
100%29.230.8960.1571.10

Results are evaluated on N3DV. The anchor ratio is measured relative to the static token set.

(2) Spatially concentrated partitions and static anchors.

We added structured-partition experiments across six scenes and both backbones (table below). Random subsets retain near-full reconstruction quality, whereas contiguous-band and foreground/background partitions degrade substantially. This consistent trend shows that spatial coverage matters, motivating sparse reference anchors to supply context beyond the locally concentrated dynamic tokens. The supplementary visualizations illustrate these partitions and the results with anchors.

Table 6 above directly validates this design: adding 10% static anchors improves PSNR from 28.48 to 29.12 dB, while increasing latency from 0.29 to 0.30 s/frame. Larger ratios yield diminishing quality gains at higher cost. Together, these results connect the random-partition observation in Fig. 2 to our practical update mechanism: local dynamic reconstruction supported by sparse reference static context.

TTT-LRM

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini20.7019.6821.5321.19
cook_spinach25.9428.0230.2430.55
flame_salmon20.2920.3221.7021.36
cut_roasted_beef25.0926.7229.3429.54
flame_steak24.7728.3830.6030.70
sear_steak23.9228.2530.4830.67
Average23.4525.2327.3227.34

WorldMirror

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini17.9018.5419.3619.15
cook_spinach24.3024.9328.3328.35
flame_salmon19.4018.1319.4819.46
cut_roasted_beef21.7325.1927.4027.30
flame_steak23.0125.0928.4128.51
sear_steak21.3826.7428.4628.53
Average21.2923.1025.2425.22

W2. Separating background stabilization from dynamic-content preservation and clarifying residual maps.

We thank the reviewer for highlighting that full-image tOF should be complemented by regional evaluation to identify where the benefits arise. We therefore added regional reconstruction metrics and temporal evaluation within person ROIs.

(1) Regional reconstruction quality.

Average dynamic-region PSNR decreases from 26.27 dB for TTT-LRM to 26.13 dB for ReStream, a reduction of 0.14 dB, with per-scene decreases ranging from 0.08 to 0.22 dB. Static-region PSNR decreases from 30.45 to 30.38 dB. These results show that the quality cost remains small within dynamic regions, rather than only in full-image averages dominated by static backgrounds.

SceneTTT-LRM dyn/staOurs dyn/staΔdyn
coffee_martini23.18 / 27.5423.10 / 27.61-0.08
cook_spinach27.41 / 31.8327.32 / 31.71-0.09
cut_roasted_beef26.80 / 30.5726.65 / 30.46-0.15
flame_salmon24.22 / 28.8024.08 / 28.73-0.14
flame_steak27.70 / 32.2827.54 / 32.22-0.16
sear_steak28.32 / 31.6928.10 / 31.56-0.22
Mean26.27 / 30.4526.13 / 30.38-0.14

PSNR values are in dB. dyn/sta denote dynamic/static regions. Δdyn = Ours − TTT-LRM.

Regional SSIM and LPIPS provide complementary evidence: dynamic-region SSIM increases from 0.880 to 0.887, while LPIPS increases slightly from 0.143 to 0.146. Both static-region metrics change by only 0.001. Together, these results quantify the quality trade-off introduced by selective updates.

RegionSSIM ↑: TTT-LRM / OursLPIPS ↓: TTT-LRM / Ours
Full image0.896 / 0.8950.157 / 0.158
Dynamic0.880 / 0.8870.143 / 0.146
Static0.896 / 0.8950.158 / 0.159

(2) Temporal performance on moving subjects.

tOF measures the consistency of reconstructed motion with ground-truth motion. Suppressing background flicker from independent frame-by-frame reconstruction is one of ReStream’s objectives, so the full-image tOF improvement includes a practically meaningful background-stabilization benefit.

To assess moving subjects, we compute tOF within the same person ROIs across methods. ReStream and frame-by-frame TTT-LRM obtain 0.257 and 0.259, respectively, indicating comparable temporal performance within these ROIs. Together with the full-image results, this supports ReStream’s ability to reduce background flicker while preserving temporal performance on moving subjects.

(3) Definition of the residual maps in Fig. 6.

Fig. 6 visualizes the absolute difference between the current reconstruction and the reference-frame reconstruction at timestamp 0: Rt=∣I^t−I^0∣R_t = |\hat{I}_t-\hat{I}_0|. It depicts the spatial distribution of appearance changes relative to the reference frame.

In the displayed static regions, ReStream exhibits smaller appearance fluctuations than frame-by-frame TTT-LRM, illustrating the stabilizing effect of static reuse. Within dynamic regions, however, these differences also include genuine motion and visibility changes, so their magnitude is not a direct measure of reconstruction error. Dynamic-content preservation is instead supported by the regional quality metrics and person-ROI tOF reported above.

W3. Layer-wise caching, normalization, residual connections, and TTT fast-weight initialization.

We provide the following implementation details to clarify the relationship between reference caches, current-frame computation, and TTT fast weights.

(1) Cached tensors at each layer.

Reference caches are initialized at frame 0. Cached submodule outputs are taken before residual addition:

  • SA: We cache outputs and K/V. At subsequent frames, Q, K, and V are recomputed for tokens participating in the update; the remaining static positions reuse reference K/V and outputs.
  • TTT: We cache outputs for all tokens. Subsequent interactions operate on dynamic tokens and sampled static anchors, while other static positions reuse reference outputs.
  • MLP: We cache outputs for reuse at static positions.
  • Before decoding: All static positions, including sampled anchors, are replaced with the corresponding final-layer token representations from frame 0.

(2) Normalization and residual connections.

LN, QK-norm, and o_norm operate independently per token, without aggregating statistics across tokens. Selective updates therefore do not alter other tokens’ normalization statistics.

We cache submodule outputs, not the residual stream itself. Residual connections follow the original architecture, so intermediate static representations may combine the current residual stream with cached reference outputs. Explicit replacement before decoding ensures that the final static tokens match their reference-frame counterparts.

(3) Frame-dependent masks.

The reference cache contains results for all tokens, indexed by view and patch position. Each frame’s mask determines which positions are updated and which read from the cache; changing masks therefore do not require rebuilding the reference cache. Subsequent frames share the reference state rather than propagating updated results from one frame to the next.

We randomly sample 10% of the current static tokens as anchors to provide context for dynamic interaction. Their updated outputs are not retained as final static representations: these positions are also restored to their reference values before decoding.

(4) WinitW_{\mathrm{init}} in Eq. (9) and reference fast weights.

WinitW_{\mathrm{init}} is the fast-weight initialization learned by the pretrained TTT-LRM, not the weights adapted on the reference frame. We distinguish it from the reference fast weights WrefW_{\mathrm{ref}}:

  • WrefW_{\mathrm{ref}} is obtained by adapting WinitW_{\mathrm{init}} on the dynamic branch during reference-frame processing and is used only in layers that perform Apply without Update.
  • Layers that perform Update start from WinitW_{\mathrm{init}} at each timestamp and produce frame-specific weights WtW_t. These weights are not propagated to the next timestamp.

Thus, reference-token output reuse, reference fast-weight reuse in Apply-only layers, and per-frame adaptation in Update layers are distinct mechanisms. We will clarify them separately in the revised manuscript and release the implementation to support reproducibility.

Reviewer_EVjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReStream accelerates volumetric video reconstruction from synchronized videos captured by fixed cameras. It caches reference-frame representations, identifies changed patches using embedding differences, and updates dynamic tokens together with sparse static anchors. The strategy requires no additional training and is evaluated with TTT-LRM and WorldMirror. The main contribution is selective computation within pretrained reconstruction models.

Critical strengths

1.The paper is well written, and its motivation is clear.

2.The method is about 3.6× faster than TTT-LRM, with only a 0.11 dB drop in mean PSNR and better temporal stability. It requires no additional training and works with both tested attention- and TTT-based backbones.

Critical weaknesses

1.Limited Applicability: Since the proposed method requires distinguishing between static and dynamic tokens, the authors acknowledge that it is only applicable to scenes with fixed cameras. Compared with TTT-based reconstruction models, this is a significant limitation.

2.The decomposition experiment is limited. Figure 2 splits current-frame tokens into interleaved groups. The actual method updates spatially concentrated regions and reuses features from an earlier frame. Figure 2 alone does not show how well its observation carries over to this setting. Testing contiguous partitions and regions affected by occlusion on both backbones would make the claim more convincing.

3.The patch-difference mask needs comparison with other choices. The threshold ablation does not establish how this mask compares with optical flow combined with occlusion or visibility estimates. I would like to see this comparison using the same reference frame, with updated-token counts and total runtime reported, including the cost of mask estimation.

4.Reconstruction quality in challenging regions needs further evaluation. The method shows a small drop in quality relative to the baseline across the evaluated datasets. However, full-image averages may hide larger errors on moving objects, newly revealed backgrounds, and occlusion boundaries, particularly when static backgrounds dominate the image. Please report metrics for these regions separately and show difficult cases, including any ghosting, incorrect geometry, or missing content, to assess whether the quality loss remains acceptable under more challenging motion and visibility changes.

Questions for authors

1.Does the decomposition result in Figure 2 hold for contiguous token groups and regions affected by occlusion on both backbones?

2.How strict is the fixed-camera requirement? Can the method tolerate slight camera motion or vibration, and how would this affect reconstruction quality and speedup?

3.Can you report quality separately for moving objects, newly revealed backgrounds, and occlusion boundaries, along with examples of ghosting, incorrect geometry, or missing content?

4.How does the patch-difference mask compare with masks based on optical flow and visibility estimates? Please report updated-token counts and total runtime, including mask estimation, using the same reference frame.

5.How would the proposed method perform in outdoor scenes with significant background changes?

5 comments

We thank the reviewer EVjy for recognizing the paper’s clarity and motivation, as well as ReStream’s training-free acceleration, quality preservation, temporal stability, and applicability to both backbones.

W1. Target setting and practical applicability.

Our work targets synchronized multi-view volumetric video captured by fixed cameras—a common acquisition setting used by 3DGStream, MixVoxels, 4K4D, and FreeTimeGS, rather than a restriction introduced specifically for ReStream.

Our objective is to exploit temporal redundancy in this practical setting to accelerate pretrained reconstruction models. The general reconstruction capabilities of the original TTT backbone and the conditions enabling ReStream’s acceleration are distinct: we provide efficient, training-free streaming inference for fixed-camera volumetric video, rather than claim the same acceleration benefits under arbitrary camera motion.

W2. Decomposability under spatially structured partitions.

We agree that the interleaved or random subsets in Fig. 2 retain broad spatial coverage, whereas ReStream’s dynamic updates are typically more spatially concentrated. To examine this distinction, we added contiguous-band, foreground/background, and random partition experiments across six scenes and both backbones. Visualizations are available here.

TTT-LRM

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini20.7019.6821.5321.19
cook_spinach25.9428.0230.2430.55
flame_salmon20.2920.3221.7021.36
cut_roasted_beef25.0926.7229.3429.54
flame_steak24.7728.3830.6030.70
sear_steak23.9228.2530.4830.67
Average23.4525.2327.3227.34

WorldMirror

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini17.9018.5419.3619.15
cook_spinach24.3024.9328.3328.35
flame_salmon19.4018.1319.4819.46
cut_roasted_beef21.7325.1927.4027.30
flame_steak23.0125.0928.4128.51
sear_steak21.3826.7428.4628.53
Average21.2923.1025.2425.22

PSNR (dB), evaluated with six input views at frame 0. Random and band partitions use a 20/80 split. No split denotes full-token inference.

Both backbones show a consistent trend: random partitioning preserves quality close to full inference, while contiguous-band and foreground/background partitions substantially reduce quality. These results indicate that decomposability depends on the spatial context retained by each subset. Random sampling provides sparse coverage of the scene, whereas spatially concentrated subsets can lack geometric references from other regions.

This motivates our static-anchor design: ReStream does not require local dynamic tokens to reconstruct independently of the reference static scene. Instead, reference static anchors provide spatial context and geometric references for dynamic updates. Table 6 supports this design: adding 10% static anchors improves PSNR from 28.48 to 29.12 dB, while latency increases only from 0.29 to 0.30 s/frame.

We therefore interpret Fig. 2 as an empirical observation motivating selective interaction, rather than a guarantee under arbitrary partitions. The additional experiments establish the effect of spatial partitioning; cross-frame occlusion and disocclusion involve visibility changes beyond what these single-frame experiments alone establish.

W3. Comparison with an optical-flow-based mask.

We added an update-mask comparison using bidirectional RAFT optical flow with 12 iterations, evaluating both reconstruction quality and mask-estimation cost.

RAFT inference takes approximately 120 ms, and the complete mask-estimation pipeline, including post-processing, takes approximately 150 ms. Patch-difference mask estimation takes 6 ms. The table below includes both mask estimation and subsequent model inference. Mask visualizations are available here.

Mask methodPSNR ↑SSIM ↑LPIPS ↓Mask estimation (ms) ↓Model inference (ms) ↓Total time (ms) ↓
RAFT-based29.090.8940.159150320470
Patch difference29.120.8950.1586300306

For the RAFT-based method, mask estimation includes optical-flow inference and associated post-processing. Total time is the sum of mask-estimation and model-inference time.

Both masks achieve comparable reconstruction quality: patch difference obtains 29.12 dB PSNR, versus 29.09 dB for the RAFT-based mask. Patch difference incurs substantially less mask-estimation overhead, reducing total runtime from 470 to 306 ms. These results support our lightweight default while demonstrating that optical-flow-based masks can also be integrated as an alternative update-selection strategy.

W4. Reconstruction quality in challenging regions.

(1) Separate evaluation of dynamic and static regions.

We added regional evaluations, with the dynamic region covering moving objects and newly revealed backgrounds. With the complete ReStream configuration, average dynamic-region PSNR decreases from 26.27 to 26.13 dB, a reduction of 0.14 dB, with per-scene decreases ranging from 0.08 to 0.22 dB. Thus, the PSNR cost remains small within dynamic regions, rather than only in full-image averages.

SceneTTT-LRM dyn/staOurs dyn/staΔdyn
coffee_martini23.18 / 27.5423.10 / 27.61-0.08
cook_spinach27.41 / 31.8327.32 / 31.71-0.09
cut_roasted_beef26.80 / 30.5726.65 / 30.46-0.15
flame_salmon24.22 / 28.8024.08 / 28.73-0.14
flame_steak27.70 / 32.2827.54 / 32.22-0.16
sear_steak28.32 / 31.6928.10 / 31.56-0.22
Mean26.27 / 30.4526.13 / 30.38-0.14

PSNR values are in dB. dyn/sta denote dynamic/static regions. Δdyn = Ours − TTT-LRM. Moving objects and newly revealed backgrounds are evaluated jointly within the dynamic region.

Regional SSIM and LPIPS provide complementary evidence. Dynamic-region SSIM increases from 0.880 to 0.887, while LPIPS increases slightly from 0.143 to 0.146. Together, these metrics indicate comparable reconstruction quality within the evaluated dynamic regions.

RegionSSIM ↑: TTT-LRM / OursLPIPS ↓: TTT-LRM / Ours
Full image0.896 / 0.8950.157 / 0.158
Dynamic0.880 / 0.8870.143 / 0.146
Static0.896 / 0.8950.158 / 0.159

(2) Boundary quality and visibility changes.

We provide close-up boundary comparisons. In the displayed examples, ReStream and frame-by-frame TTT-LRM exhibit comparable boundary quality. These visualizations complement the regional metrics by exposing local reconstruction details.

We also provide video comparisons for three sequences, where the methods show comparable reconstruction of dynamic content:

Q1. Does the observation in Fig. 2 hold for contiguous partitions and occlusion-affected regions on both backbones?

The additional experiments in W2 cover both backbones and six scenes. Random partitions preserve quality close to full inference, whereas contiguous-band and foreground/background partitions substantially reduce quality. This shows that the observation depends on the spatial context retained by each subset, motivating our use of spatially distributed static anchors to provide broader scene context and reference geometry for local dynamic updates. Regional quality and boundary visualizations are provided in W4.

Q2. How strict is the fixed-camera requirement? Can ReStream tolerate slight camera motion or vibration?

ReStream does not require the same reference frame throughout an entire video. For an occasional camera displacement followed by a stable period, we can reconstruct a new reference frame and refresh the cache, allowing subsequent updates to use the new camera configuration. Thus, the fixed-camera assumption primarily applies within each cache-reuse interval, rather than requiring the initial cache to remain unchanged throughout a long sequence.

The initialization-amortization experiment in Table 5(below) shows that short reference intervals can still retain substantial acceleration: for a 20-frame sequence including one full reference reconstruction, ReStream takes 6.56 s versus 21.83 s for full frame-by-frame inference, yielding approximately 3.3× speedup. This quantifies the computational cost of reference initialization, rather than directly measuring performance under camera vibration.

Under continuous vibration, camera poses change within the reuse interval, potentially affecting cache correspondence and increasing the update budget. Its impact therefore depends on both the refresh frequency and the magnitude and persistence of camera motion within each interval.

FramesTTT-LRM total time (s) ↓Ours total time (s) ↓Ours avg. (s/frame) ↓Speedup ↑
11.091.091.0911.00×
55.462.290.4582.38×
1010.913.690.3692.95×
2021.836.560.3283.32×
5054.5715.590.3113.49×
100109.1329.990.2993.63×
300327.3988.690.2953.69×

Q3. Can you report quality separately for moving objects, newly revealed backgrounds, and occlusion boundaries?

W4 provides separate evaluations of dynamic and static regions, with moving objects and newly revealed backgrounds evaluated jointly within the dynamic region. ReStream’s average dynamic-region PSNR is only 0.14 dB below full TTT-LRM, with per-scene differences of 0.08–0.22 dB. Thus, the small quality cost is not confined to full-image averages. Fig. 9 additionally provides boundary close-ups, showing comparable boundary reconstruction in the displayed examples.

Q4. How does the patch-difference mask compare with optical-flow/visibility-based masks?

W3 compares our mask with one derived from bidirectional RAFT optical flow, with visualizations available here. In the displayed examples, the flow-based mask provides more continuous coverage of moving regions but incurs higher estimation overhead.

Reconstruction quality is comparable: RAFT-based and patch-difference masks achieve PSNRs of 29.09 and 29.12 dB, SSIMs of 0.894 and 0.895, and LPIPS scores of 0.159 and 0.158, respectively. RAFT inference alone takes approximately 120 ms, whereas patch-difference computation takes only 6 ms. Under this setting, patch difference achieves comparable quality at a substantially lower mask-estimation cost.

Q5. How does ReStream perform in outdoor scenes with substantial background changes?

Stable backgrounds provide reusable temporal redundancy for dynamic reconstruction. Like other reconstruction methods that exploit this redundancy, ReStream cannot also maintain the same acceleration benefits when the background undergoes widespread, continuous changes. Optimization-based methods that reuse temporal information require additional representation (E.g. more 3DGS points) updates or fitting for changed content, while ReStream recomputes the corresponding tokens. Both face the same underlying constraint: less reusable content means less computation can be saved.

In ReStream, background regions detected as changed participate in the current-frame update rather than retaining outdated representations. When nearly all tokens require updating, interaction costs approach full frame-by-frame inference, with additional change-detection and cache-management overhead. This reflects an efficiency boundary shared by temporal-reuse strategies, rather than a limitation unique to ReStream.

Reviewer_8mAFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReStream speeds up frame-by-frame inference of pretrained feed-forward multi-view reconstruction models on fixed-camera multi-view video. It runs the full model on a reference frame and caches per-layer token features. On later frames it recomputes only tokens whose patch embedding differs from the reference by more than τ = 0.02 (L2), plus a random 10% of static tokens as "anchors"; every other token reuses its cached features.

In frame attention, dynamic queries attend to cached static keys and values; global attention and TTT fast-weight updates run over dynamic tokens plus anchors. The design is motivated by an observation that processing two complementary token subsets independently nearly matches full processing (29.38 vs. 29.52 dB). On N3DV, MeetRoom, ZJU-SelfCap and ENeRF with six 960×720 inputs, it reports ~3.6× speedup on TTT-LRM and WorldMirror at a ~0.1 dB PSNR cost, with lower tOF.

Critical strengths

1Training-free and plug-and-play, addressing a real deployment bottleneck for reconstruction models applied to volumetric video. 2One principle covers attention and TTT backbones, and App. A.1 explains each case clearly. Caching static keys and values for frame attention is a sensible choice. 3Useful ablations: refresh interval (Table 4), threshold (Fig. 7), anchor ratio (Table 6), sequence length (Table 5), and block-level profiling (Table 2) showing where the time goes. Table 6's 100%-anchor row recovers exactly the full-model PSNR (29.23), so static reuse alone costs almost nothing. 4The temporal-consistency gain (tOF 0.282 → 0.192) is a real practical benefit.

Critical weaknesses

W1. The closest prior work is missing, and novelty is overstated. Recomputing only changed tokens and reusing cached activations for the rest is an established idea.

Eventful Transformers (Dutson et al., ICCV 2023) does this for vision transformers on video, gating tokens by change and reusing cached attention and MLP outputs. DeltaCNN (Parger et al., CVPR 2022), Skip-Convolutions (Habibian et al., CVPR 2021) and CBInfer (Cavigelli et al., 2017) do it for CNNs; feature caching is also standard in diffusion acceleration (e.g., DeepCache, ToCa). In volumetric video, StreamRF (Li et al., NeurIPS 2022) — cited here only as the source of MeetRoom — and 3DGStream (Sun et al., CVPR 2024) update only changing regions frame to frame. The contribution should be framed as adapting token reuse to cross-view reconstruction models, where the new questions are global attention, TTT and anchoring. It should be compared with an Eventful-style policy (top-k change gating; reference- vs. previous-frame updates). W2. The decomposability observation does not match the regime the method runs in. Fig. 2 partitions tokens randomly, so each subset still spans every view and the whole image at lower density. That is robustness to random token subsampling, which is expected for redundant multi-view tokens and for TTT updates that average over tokens. ReStream instead processes a spatially clustered dynamic set with only 10% random static context. The observation is also shown on one scene with one model; please test spatially structured splits (contiguous blocks, foreground vs. background), more scenes and both backbones.

W3. The metrics are dominated by background that is copied, not reconstructed. With roughly 80–90% of tokens reused verbatim, full-image PSNR, SSIM and LPIPS largely score background taken from frame 0. Quality on the moving content — the point of a dynamic reconstructor — is not measured. Table 6 suggests the dynamic region is sensitive: removing anchors costs 0.64 dB over the whole image, which could be several dB on the subject itself.

Report metrics masked to dynamic regions (ground-truth foreground masks, or the dilated dynamic mask). Show zoomed results at the boundary between reused and recomputed tokens, where seams or ghosting at disocclusions would appear. Report tOF restricted to dynamic regions too; frozen static regions lower tOF mechanically. W4. The evaluation regime is favorable, and the baselines do not test the claim.

All four datasets have a mostly static background with one or a few actors. There are no crowded or fast scenes, no global illumination change and no auto-exposure flicker, where most tokens would be flagged. §5 concedes cost then approaches full inference, but no curve of speedup and quality against dynamic fraction is shown; Fig. 7 uses two scenes. DepthSplat and WorldMirror at lower resolution (Table 1, Fig. 4) do not test an acceleration method. DepthSplat's ENeRF PSNR of 10.79 also suggests a misconfiguration. Missing are equal-latency baselines: the backbone at lower resolution or with fewer views at ~0.3 s; random token dropping at the same token budget; previous-frame instead of reference-frame reuse; an Eventful-style gate. The dynamic feed-forward methods named in related work (SplatVoxel, Nopo4D) and the unnamed chunk-wise method in Fig. 1 are not compared. Anchors are random, yet no variance across seeds is reported. Are anchors resampled each frame, which could cause jitter, or fixed? W5. Several method and protocol details are unclear.

τ = 0.02 on the raw L2 distance between patch embeddings is shared across TTT-LRM and WorldMirror, whose embedding scales differ. Is Δ normalized? How sensitive is it to sensor noise and compression? Notation mixes X_{0,S} with X_{t,S} and X_{0,A} with X_{t,A} (Eq. 3 and text). The static set is defined per frame but its features come from the reference. Do anchors at layer ℓ use cached layer-ℓ reference features? Why random anchors for global attention, instead of letting dynamic queries attend to all cached static keys and values as in frame attention? That costs O(N_D·N), still far below O(N²), and an ablation would show what anchoring gives up. The complexity analysis also drops the anchor term: with anchors, global attention is O((N_D + ηN_S)²), not O(N_D²). For TTT, fast weights fit on dynamic tokens plus 10% anchors see a very different token distribution than in training. Some discussion of why this works, and when it would not, is warranted. Memory for the per-layer caches (plus static keys and values) is not reported. At six 960×720 views with a deep backbone this may reach gigabytes, which matters because memory is the stated weakness of chunk-wise methods. Is camera 0, the evaluation target, excluded from the six "uniformly sampled" inputs? If not, the metrics measure input-view reconstruction rather than novel-view synthesis. How many scenes per dataset are used?

Questions for authors

1 What are PSNR, SSIM, LPIPS and tOF restricted to dynamic regions? 2 How does ReStream compare with an Eventful-Transformer-style gate and with equal-latency baselines? 3 Is the target view among the inputs? 4 What is the cache memory overhead? 5 How do speedup and quality vary with the fraction of dynamic tokens across scenes? 6 Are anchors fixed or resampled, and what is the variance over seeds?

5 comments

We thank the reviewer 8mAF for the careful review and constructive feedback, and for recognizing ReStream’s practical value as a training-free, plug-and-play framework, its applicability to both attention- and TTT-based backbones, and the evidence provided by our ablations and temporal-consistency improvements. Below, we address your concerns with additional experiments and implementation details.

W1. Relation to prior work and contribution scope.

We agree that change-driven selective computation and activation caching are well established. ReStream’s contribution lies in how dynamic content is reconstructed with sparse reference context: rather than only selecting tokens to recompute, it restricts cross-view interaction to dynamic tokens and a small set of reference static anchors. These anchors provide scene context for dynamic reconstruction, while static outputs retain their reference representations. Thus, tokens that provide context need not have their updated outputs retained. We implement this principle in both global attention and TTT fast-weight updates, while preserving full cached K/V context within each view. We have expanded the related-work discussion and highlighted the revisions in orange.

StreamRF and 3DGStream update scene representations through scene-specific online optimization. ReStream instead reuses intermediate representations during pretrained feed-forward inference, without additional scene-specific optimization.

We share the principle of change-driven selective computation with Eventful Transformers. The key distinction is how global context is processed: Eventful incrementally updates cached attention computations, whereas ReStream restricts cross-view interaction to dynamic tokens and sparse reference static anchors. These anchors provide geometric context while static outputs retain their reference representations.

AspectEventful TransformersReStream
Global interactionIncremental updates to cached attention computationsReduced interaction set: dynamic tokens + static anchors
Reference statePer-token state from its most recent updateReference-frame features within each refresh interval
Context handlingBuffered token and attention statesframe-attention: Full static K/V within views; global-interaction: sparse anchors across views
Demonstrated mechanismsStandard attentionAttention and TTT, including selective fast-weight Update

ReStream thus addresses how to retain reconstruction context while reducing global interaction, rather than proposing token gating itself. Table 6 supports the anchor design: 10% static anchors improve PSNR by 0.64 dB at an additional 0.01 s/frame.

Anchor RatioPSNR ↑SSIM ↑LPIPS ↓Time (s/frame) ↓
0%28.480.8820.1670.29
5%28.960.8920.1600.30
10%29.120.8950.1580.30
20%29.170.8950.1580.35
100%29.230.8960.1571.10

W2. Relationship between token decomposability and the actual update setting.

We agree that the random subsets in Fig. 2 typically retain broad spatial coverage. The result should therefore be interpreted as an empirical observation under random partitioning, rather than a guarantee of decomposability under arbitrary spatial partitions.

Following your suggestion, we added contiguous-band, foreground/background, and random partition experiments across six scenes and both TTT-LRM and WorldMirror. Both backbones exhibit a consistent trend: random partitioning achieves average PSNRs of 27.32 and 25.24 dB, respectively, close to full inference at 27.34 and 25.22 dB. In contrast, contiguous-band partitioning yields 23.45 and 21.29 dB, while foreground/background partitioning yields 25.23 and 23.10 dB.

These results show that the random-partition observation in Fig. 2 does not directly generalize to spatially concentrated subsets, supporting our motivation for static anchors. ReStream does not rely on reconstructing local dynamic tokens entirely independently. Instead, it lets them interact with a small set of reference static anchors to provide spatial context and geometric references. Table 6 further supports this design: adding 10% static anchors improves PSNR from 28.48 to 29.12 dB, while latency increases only from 0.29 to 0.30 s/frame.

Visualizations of the partitions and reconstruction results are available in the supplementary material.

TTT-LRM

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini20.7019.6821.5321.19
cook_spinach25.9428.0230.2430.55
flame_salmon20.2920.3221.7021.36
cut_roasted_beef25.0926.7229.3429.54
flame_steak24.7728.3830.6030.70
sear_steak23.9228.2530.4830.67
Average23.4525.2327.3227.34

PSNR (dB), evaluated using TTT-LRM with six input views at frame 0. Random and band partitions use a 20/80 split. No split denotes full-token inference.

WorldMirror

SceneBand (20/80)Foreground / BackgroundRandom (20/80)No split
coffee_martini17.9018.5419.3619.15
cook_spinach24.3024.9328.3328.35
flame_salmon19.4018.1319.4819.46
cut_roasted_beef21.7325.1927.4027.30
flame_steak23.0125.0928.4128.51
sear_steak21.3826.7428.4628.53
Average21.2923.1025.2425.22

PSNR (dB), evaluated using WorldMirror with six input views at frame 0. Random and band partitions use a 20/80 split. No split denotes full-token inference.

W3. Dynamic-region quality and temporal consistency.

(1) Separate evaluation of dynamic and static regions.

We report reconstruction quality separately for dynamic and static regions. With the complete ReStream configuration, dynamic-region PSNR decreases from 26.27 to 26.13 dB, a reduction of 0.14 dB. Across the six scenes, the decrease ranges from 0.08 to 0.22 dB. These results show that the PSNR cost remains small within dynamic regions, rather than only in full-image averages dominated by static backgrounds.

SceneTTT-LRM dyn/staOurs dyn/staΔdyn
coffee_martini23.18 / 27.5423.10 / 27.61-0.08
cook_spinach27.41 / 31.8327.32 / 31.71-0.09
cut_roasted_beef26.80 / 30.5726.65 / 30.46-0.15
flame_salmon24.22 / 28.8024.08 / 28.73-0.14
flame_steak27.70 / 32.2827.54 / 32.22-0.16
sear_steak28.32 / 31.6928.10 / 31.56-0.22
Mean26.27 / 30.4526.13 / 30.38-0.14

PSNR values are in dB. dyn/sta denote dynamic/static regions. Δdyn = Ours − TTT-LRM.

Regional SSIM and LPIPS provide complementary evidence: dynamic-region SSIM increases from 0.880 to 0.887, while LPIPS increases slightly from 0.143 to 0.146.

RegionSSIM ↑: TTT-LRM / OursLPIPS ↓: TTT-LRM / Ours
Full image0.896 / 0.8950.157 / 0.158
Dynamic0.880 / 0.8870.143 / 0.146
Static0.896 / 0.8950.158 / 0.159

(2) Temporal consistency within person ROIs.

tOF evaluates the consistency of reconstructed motion with ground-truth motion. Suppressing background flicker caused by independent frame-by-frame reconstruction is one of ReStream’s objectives, so the reduction in full-image tOF includes a practically meaningful background-stabilization benefit.

To additionally assess moving content, we compute tOF within the same person ROIs across methods. ReStream and frame-by-frame TTT-LRM obtain 0.257 and 0.259, respectively, indicating comparable temporal performance within these ROIs. Together with the full-image results, this supports ReStream’s ability to stabilize the background while maintaining temporal fidelity on moving subjects.

(3) Static anchors and boundary quality.

Table 6 shows that removing static anchors reduces reconstruction quality, supporting their role in providing reference geometric context for spatially concentrated dynamic tokens. All regional results above use the default 10% static-anchor ratio and therefore evaluate the complete method.

We also provide close-up boundary comparisons. In the displayed examples, ReStream and TTT-LRM exhibit comparable reconstruction quality at the boundaries between reused and updated regions.

Comments on the paper

1 comment

Overall Response

We thank all three reviewers for their time and thoughtful reviews, and for recognizing ReStream’s novelty (KEoz) , clear motivation (EVjy) and presentation, and practical (KEoz, EVjy, 8mAF) , training-free acceleration across attention and TTT backbones (KEoz, EVjy, 8mAF) . Below, we address the shared concerns about novelty, spatial context, dynamic-content preservation, evaluation, and implementation through additional experiments and clarifications.

1. Contribution: sparse reference context for selective multi-view reconstruction.(KEo 8mAF)

ReStream’s contribution is a selective reconstruction mechanism that uses dynamic tokens and sparse reference static anchors to reconstruct changing content consistently with the cached static scene. It preserves full static K/V within each view, restricts global cross-view interaction and TTT fast-weight updates to dynamic tokens plus anchors, and retains reference representations at static positions. Thus, it distinguishes tokens participating in contextual interaction from positions whose updates are retained, going beyond selecting which tokens to recompute.

2. Evidence for spatial context and static anchoring.(KEoz EVjy 8mAF)

New experiments across six scenes and both backbones show that random partitions preserve quality close to full inference, whereas contiguous and foreground/background partitions degrade reconstruction. This clarifies Fig. 2 as a context-dependent empirical observation and motivates static anchoring. The anchor ablation provides direct evidence: adding 10% static anchors improves PSNR by 0.64 dB, while increasing latency by only 0.01 s/frame.

3. Dynamic-content preservation and temporal evaluation.(KEoz EVjy 8mAF)

We added separate dynamic/static reconstruction metrics. Dynamic-region PSNR decreases by 0.14 dB on average, with per-scene decreases of 0.08–0.22 dB. Person-ROI tOF is 0.257 for ReStream versus 0.259 for TTT-LRM, indicating comparable temporal performance on the evaluated moving subjects. Boundary close-ups and video comparisons complement these regional measurements.

We also clarify that Fig. 6 shows absolute image differences from the reference-frame reconstruction, not a temporal-error map. It illustrates background stabilization; dynamic fidelity is assessed through reconstruction metrics and motion-referenced tOF.

4. Additional efficiency comparisons and evaluation scope.(EVjy)

At approximately 0.30 s/frame, ReStream achieves 29.12 dB, compared with 24.36 dB for lower-resolution TTT-LRM. A bidirectional RAFT-mask comparison achieves similar reconstruction quality but incurs higher total runtime: 470 ms versus 306 ms for patch difference, including mask estimation. Update-budget experiments further characterize the quality–latency trade-off.

We clarify that ReStream targets fixed-camera multi-view video and exploits the temporal redundancy available in that setting. As more content requires updating, the acceleration benefit decreases. We distinguish this operating scope from the broader reconstruction capabilities of the underlying backbone.

5. Reproducibility and protocol clarifications.(KEoz 8mAF)

We specify the cached tensors, their positions relative to normalization and residual connections, changing-mask behavior, and the distinction between learned TTT initialization and reference-adapted fast weights. We also correct the complexity analysis to include anchors and clarify target-view exclusion and timing boundaries.

Three-seed evaluations yield a PSNR sample standard deviation of 0.032 dB, supporting sampling stability under the tested settings. Finally, after rechecking camera selection, we corrected DepthSplat’s ENeRF PSNR from 10.79 to 21.84 dB.

Together, these results strengthen the evidence that ReStream preserves dynamic reconstruction quality while reducing redundant computation, and make its contribution, implementation, and operating conditions more explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.