- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ReStream, a training-free approach to accelerating volumetric video reconstruction from synchronized, fixed-view cameras. It caches representations from a reference frame, identifies changed patches using embedding differences, and updates dynamic tokens together with a small subset of static anchors. The approach supports both attention-based and test-time-training-based reconstruction backbones.
The reported results demonstrate a promising efficiency–quality trade-off: approximately 3.6× acceleration over TTT-LRM, with mean PSNR decreasing from 26.02 to 25.91 dB and temporal optical-flow error decreasing from 0.282 to 0.192. The main contribution is a practical adaptation of temporal computation reuse to multi-view reconstruction. The principal questions concern its contribution beyond established token-reuse methods and whether the evaluation sufficiently establishes preservation of dynamic content.
Critical strengths
-
Technical novelty and innovation: The combination of reference-frame caching, selective dynamic updates, and static anchors is simple and technically plausible. Applying the strategy to both attention and TTT interaction mechanisms is a useful contribution, particularly because it requires no additional model training.
-
Experimental rigor and validation: The matched-backbone comparisons are the strongest evidence. Appendix A.4 specifies identical pretrained weights, input views, preprocessing, and evaluation frames. Experiments cover four datasets, and the paper includes ablations on the change threshold, reference-refresh interval, and anchor ratio. Table 4 explicitly includes initialization and refresh costs.
-
Clarity of presentation: The overall pipeline is easy to understand, and Figure 3 communicates the computation-reuse mechanism effectively. Appendix A.1 usefully distinguishes frame attention, global attention, and TTT updates.
-
Significance of contributions: A roughly 3.6× speedup with a small average reconstruction-quality reduction would be valuable to practitioners. The additional results on WorldMirror suggest that the approach is not restricted to one interaction architecture.
Critical weaknesses
W1. The methodological contribution relative to existing temporal token reuse is insufficiently established.
Eventful Transformers, ICCV 2023 already identifies changed tokens, reuses previous computations, and selectively updates transformer operations, generally without retraining. This does not establish equivalence to ReStream: multi-view geometry, static anchoring, and TTT adaptation may introduce meaningful differences. However, the paper does not explain these differences sufficiently to establish what new methodological insight enables its results.
The patch-level decomposability experiment does not fully resolve this concern. Independently processing complementary subsets of the same frame differs from combining current dynamic representations with cached representations computed under an earlier scene configuration. Moreover, spatially distributed random subsets can preserve scene-wide context, whereas a dynamic mask may be concentrated around a moving object. Figure 2 therefore supports a narrower observation than the general justification presented in the text.
W2. The temporal evaluation does not sufficiently separate background stabilization from preservation of dynamic content.
The reduction in tOF is encouraging, but both tOF and the reconstruction metrics are averaged over the image. Since the method explicitly reuses a large static region, improvements there could outweigh degradation around moving objects or occlusion boundaries.
This does not make tOF inappropriate: it compares reconstructed motion against ground-truth motion. The concern is that its aggregate value cannot establish where the improvement occurs. A breakdown of existing predictions into independently defined static and dynamic regions would directly test the paper’s claim that selective reconstruction preserves dynamic scene changes.
Figure 6 also needs a precise definition of its residual maps. If these show unaligned image differences, lower residuals alone do not establish better temporal fidelity, because genuine motion should produce nonzero differences.
W3. Several implementation ambiguities prevent a confident interpretation of the selective-update algorithm.
The paper does not sufficiently specify which intermediate representations are cached at each layer or how those caches interact with normalization, residual connections, and changing masks. For TTT, Equation (9) uses , but its relationship to the learned initialization and reference-frame fast weights needs clarification.
Questions for authors
-
What distinguishes ReStream from applying established temporal token gating and caching to a reconstruction backbone? Does the decomposability observation hold for the spatially concentrated masks actually produced by your method?
-
Can you report reconstruction and temporal errors separately for static and dynamic regions on representative existing evaluation sequences, using an independent mask? What precisely do the residual maps in Figure 6 measure?
-
What is in Equation (9), and which tensors are cached at each layer? Is the 100% anchor row in Table 6 an actual selective-update run or the original full-inference baseline?
-
Is camera 0 always excluded from the input views? What evaluation resolution and optical-flow estimator are used across methods, and which pipeline components are included in the main latency measurements?
5 comments
W2. Separating background stabilization from dynamic-content preservation and clarifying residual maps.
We thank the reviewer for highlighting that full-image tOF should be complemented by regional evaluation to identify where the benefits arise. We therefore added regional reconstruction metrics and temporal evaluation within person ROIs.
(1) Regional reconstruction quality.
Average dynamic-region PSNR decreases from 26.27 dB for TTT-LRM to 26.13 dB for ReStream, a reduction of 0.14 dB, with per-scene decreases ranging from 0.08 to 0.22 dB. Static-region PSNR decreases from 30.45 to 30.38 dB. These results show that the quality cost remains small within dynamic regions, rather than only in full-image averages dominated by static backgrounds.
| Scene | TTT-LRM dyn/sta | Ours dyn/sta | Δdyn |
|---|---|---|---|
| coffee_martini | 23.18 / 27.54 | 23.10 / 27.61 | -0.08 |
| cook_spinach | 27.41 / 31.83 | 27.32 / 31.71 | -0.09 |
| cut_roasted_beef | 26.80 / 30.57 | 26.65 / 30.46 | -0.15 |
| flame_salmon | 24.22 / 28.80 | 24.08 / 28.73 | -0.14 |
| flame_steak | 27.70 / 32.28 | 27.54 / 32.22 | -0.16 |
| sear_steak | 28.32 / 31.69 | 28.10 / 31.56 | -0.22 |
| Mean | 26.27 / 30.45 | 26.13 / 30.38 | -0.14 |
PSNR values are in dB. dyn/sta denote dynamic/static regions. Δdyn = Ours − TTT-LRM.
Regional SSIM and LPIPS provide complementary evidence: dynamic-region SSIM increases from 0.880 to 0.887, while LPIPS increases slightly from 0.143 to 0.146. Both static-region metrics change by only 0.001. Together, these results quantify the quality trade-off introduced by selective updates.
| Region | SSIM ↑: TTT-LRM / Ours | LPIPS ↓: TTT-LRM / Ours |
|---|---|---|
| Full image | 0.896 / 0.895 | 0.157 / 0.158 |
| Dynamic | 0.880 / 0.887 | 0.143 / 0.146 |
| Static | 0.896 / 0.895 | 0.158 / 0.159 |
(2) Temporal performance on moving subjects.
tOF measures the consistency of reconstructed motion with ground-truth motion. Suppressing background flicker from independent frame-by-frame reconstruction is one of ReStream’s objectives, so the full-image tOF improvement includes a practically meaningful background-stabilization benefit.
To assess moving subjects, we compute tOF within the same person ROIs across methods. ReStream and frame-by-frame TTT-LRM obtain 0.257 and 0.259, respectively, indicating comparable temporal performance within these ROIs. Together with the full-image results, this supports ReStream’s ability to reduce background flicker while preserving temporal performance on moving subjects.
(3) Definition of the residual maps in Fig. 6.
Fig. 6 visualizes the absolute difference between the current reconstruction and the reference-frame reconstruction at timestamp 0: . It depicts the spatial distribution of appearance changes relative to the reference frame.
In the displayed static regions, ReStream exhibits smaller appearance fluctuations than frame-by-frame TTT-LRM, illustrating the stabilizing effect of static reuse. Within dynamic regions, however, these differences also include genuine motion and visibility changes, so their magnitude is not a direct measure of reconstruction error. Dynamic-content preservation is instead supported by the regional quality metrics and person-ROI tOF reported above.
W3. Layer-wise caching, normalization, residual connections, and TTT fast-weight initialization.
We provide the following implementation details to clarify the relationship between reference caches, current-frame computation, and TTT fast weights.
(1) Cached tensors at each layer.
Reference caches are initialized at frame 0. Cached submodule outputs are taken before residual addition:
- SA: We cache outputs and K/V. At subsequent frames, Q, K, and V are recomputed for tokens participating in the update; the remaining static positions reuse reference K/V and outputs.
- TTT: We cache outputs for all tokens. Subsequent interactions operate on dynamic tokens and sampled static anchors, while other static positions reuse reference outputs.
- MLP: We cache outputs for reuse at static positions.
- Before decoding: All static positions, including sampled anchors, are replaced with the corresponding final-layer token representations from frame 0.
(2) Normalization and residual connections.
LN, QK-norm, and o_norm operate independently per token, without aggregating statistics across tokens. Selective updates therefore do not alter other tokens’ normalization statistics.
We cache submodule outputs, not the residual stream itself. Residual connections follow the original architecture, so intermediate static representations may combine the current residual stream with cached reference outputs. Explicit replacement before decoding ensures that the final static tokens match their reference-frame counterparts.
(3) Frame-dependent masks.
The reference cache contains results for all tokens, indexed by view and patch position. Each frame’s mask determines which positions are updated and which read from the cache; changing masks therefore do not require rebuilding the reference cache. Subsequent frames share the reference state rather than propagating updated results from one frame to the next.
We randomly sample 10% of the current static tokens as anchors to provide context for dynamic interaction. Their updated outputs are not retained as final static representations: these positions are also restored to their reference values before decoding.
(4) in Eq. (9) and reference fast weights.
is the fast-weight initialization learned by the pretrained TTT-LRM, not the weights adapted on the reference frame. We distinguish it from the reference fast weights :
- is obtained by adapting on the dynamic branch during reference-frame processing and is used only in layers that perform Apply without Update.
- Layers that perform Update start from at each timestamp and produce frame-specific weights . These weights are not propagated to the next timestamp.
Thus, reference-token output reuse, reference fast-weight reuse in Apply-only layers, and per-frame adaptation in Update layers are distinct mechanisms. We will clarify them separately in the revised manuscript and release the implementation to support reproducibility.
We thank the reviewer
KEozfor recognizing ReStream’s practical value, training-free applicability across attention and TTT backbones, matched-backbone evaluations, and clear presentation. Below, we address your concerns with additional evidence and clarifications.(1) Relationship to Eventful Transformers.
We agree that token gating, computation reuse, and training-free selective updates are established by Eventful Transformers. Our methodological insight is to separate the static context needed for dynamic reconstruction from the static representations that need updating. Sparse reference static anchors provide scene context for dynamic reconstruction, while final static representations remain unchanged.
Eventful Transformers reduces repeated attention computation through gating, buffering, and incremental updates. ReStream instead excludes most static tokens from current-frame cross-view interaction, retaining sparse anchors to support geometric consistency with the cached static scene. This distinction is summarized below.
For TTT, restricting the tokens participating in Update reduces fast-weight adaptation work itself; merely reducing queries in Apply would not achieve this. Together, these designs specify how to preserve reconstruction context while reducing global computation, beyond choosing which tokens to recompute.
Table 6 directly supports the role of sparse reference context: adding 10% static anchors improves PSNR by 0.64 dB, reaching 29.12 dB compared with the full model’s 29.23 dB, at only 0.01 s/frame additional cost.
Results are evaluated on N3DV. The anchor ratio is measured relative to the static token set.
(2) Spatially concentrated partitions and static anchors.
We added structured-partition experiments across six scenes and both backbones (table below). Random subsets retain near-full reconstruction quality, whereas contiguous-band and foreground/background partitions degrade substantially. This consistent trend shows that spatial coverage matters, motivating sparse reference anchors to supply context beyond the locally concentrated dynamic tokens. The supplementary visualizations illustrate these partitions and the results with anchors.
Table 6 above directly validates this design: adding 10% static anchors improves PSNR from 28.48 to 29.12 dB, while increasing latency from 0.29 to 0.30 s/frame. Larger ratios yield diminishing quality gains at higher cost. Together, these results connect the random-partition observation in Fig. 2 to our practical update mechanism: local dynamic reconstruction supported by sparse reference static context.
TTT-LRM
WorldMirror