acceptodds
Withdrawn from ICLR 2027

TTT4D: Mining Motion Cues in Test-Time Training for Scalable 4D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing dynamic 4D scenes from monocular videos remains challenging because moving objects violate the static scene assumptions of most 3D foundation models and interfere with camera and geometry estimation. Existing 4D methods often rely on fine tuning with limited dynamic data or external priors, while recent training-free approaches (e.g., Easi3R and VGGT4D) remain constrained by limited temporal scalability. In this work, we introduce TTT4D, a training-free framework that adapts the stateful 3D foundation model ZipMap to dynamic 4D reconstruction through test-time training. We observe that independent motion manifests as complementary inconsistencies in both ZipMap's internal representations and its online adaptation dynamics. Individual signals, however, are not motion specific and may also respond to static ambiguities such as newly observed regions, rich appearance, or uncertain geometry. Crucially, these ambiguities affect different internal signals differently, whereas independently moving regions exhibit consistent cross cue responses. Based on this observation, we extract complementary adaptation- and representation-level cues and fuse them to suppress ambiguous static responses and highlight coherent dynamic regions. The resulting masks guide a shallow attention routing strategy that selectively isolates dynamic features to reduce dynamic interference without retraining the pretrained model. TTT4D achieves strong performance in dynamic segmentation, camera pose estimation, and dense 4D reconstruction while retaining the efficiency and scalability of stateful feed-forward architectures, with speedups over VGGT-4D and over Easi3R on TUM-Dynamics / ADT benchmarks, respectively, using a single NVIDIA RTX PRO 6000 (96GB) GPU. Code will be released to contribute to the community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Foundation Models, 4D Scene Reconstruction, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_nhcGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes to discover motion cue (dynamic object regions) from streaming 3D reconstruction method that uses TTT (test time training), ie ZipMap. Similar to Easi3R, which recovers dynamic object regions from feedforward 3D reconstruction model, the paper similarly shows how to recover dynamic object regions by probing cues from intermediate layers and gates from TTT layers.

The paper demonstrates that the extracted masks serve as moving object segmentation as a zero-shot approach. Also the masks can be used for suppressing dynamic region and improving 3D reconstruction and camera pose accuracy.

However, due to several weaknesses (unclear intuition/insights, unjustified gains, missing recent methods for comparison, unclear details), I would rate the paper as Weak Rejection.

Critical strengths
  • Adopting Easi3R-like approach for TTT-based 3D reconstruction

    The paper shows that the Easi3R-like approach (discovering dynamic object mask from 3D reconstruction method) can be applicable to the TTT-based 3D reconstruction method which has its distinctive modules and behavior from it at read and write gate. In Fig. 3 and equation (2) shows that the writing operation involves with differentiating dynamics region from static region, ie uu.

  • Good downstream accuracy

    Empirically, the method shows overall good pose, 3D reconstruction, and moving object segmentation accuracy, compared with previous methods.

Critical weaknesses
  • Lacking intuition and insights

    The proposed formula to detect dynamic objects (ie, from Eq. (2) to Eq. (8)) are highly empirical. I am wondering how the equations are derived and what intuitions/insights are behind the equation. The core motivation share the same as Easi3R: discovering dynamic object mask from internal feature maps. I wonder what new lessons we can get from the paper, apart from the similar principle is applicable to TTT-based models. It will be interesting to analyze the TTT layers behavior on static vs dynamic region, eg what information is propagated via TTT layers. One could quantitatively measure the learning rate and gate values from write/read on both static and dynamic region. However those analyses are missing.

  • Good downstream accuracy is possibly inherited from the strong baseline model

    The strong downstream performance for dynamic scenes is somewhat expected. ZipMap is already a very strong dynamic 3D reconstruction model as shown in its website. It already has the capability for handling dynamic scenes, by using 6 dynamic scene training datasets (BEDLAM, Dynamic Replica, Kubric, OmniWorld, PointOdyssey, and Spring). Unlike Easi3R, which demonstrates discovering dynamic region information from DUSt3R (strictly trained for 3D reconstruction), This diminishes the contribution of the paper.

    Also, to be strictly fair, it's recommend to include in-depth qualitative & quantitative comparison with ZipMap as well (eg, Fig. 1, Fig. 5, and Fig. 6). According to Table 2 and 3, the accuracy improvement is marginal.

  • Missing recent results on benchmark tables (Table 2, 3)

    Many published papers are missing for comparison, e.g, MapAnything, Pi-3, St4RTrack, D4RT, Any4D, V-DPM, VGGT-Omega, FastVGGT. It is recommended to include those methods for fair comparison.

  • Unclear details and intuitions for 'dynamic interference reduction'

    The paragraph from Line 314 explains how the method utilizes the motion cues for improving downstream accuracy. However, its underlying assumptions and details are unclear.

    • I am curious what the motivation for suppressing dynamic interference is. ZipMap is already trained to reconstruct dynamic scenes in 3D. It's curious why we need to suppress it and in which cases it helps. Wouldn't it hurt the reconstruction accuracy on dynamic object regions?

    • What's the technical details on temporal stabilization (L323), smoothing translation, and gated rotation averaging on SO(3)? It will be great if further details are provided. It would be curious to see the performance without them. Also when applying these to the ZipMap baseline, does it improve their accuracy as well?

Questions for authors
  • Definition of 4D reconstruction

    I wonder what the clear definition of 4D reconstruction is. Per-frame 3D reconstruction of dynamic scene can be just considered as dynamic 3D reconstruction. 4D reconstruction may imply 3D reconstruction with 3D motion estimation. It's curious how the paper interprets the meaning of 4D reconstruction. Also, it's curious if ZipMap is strictly considered a 3D reconstruction method from the paper's perspective.

  • Confusing words -- the meaning of some terms are not clear.

    • L50: stateful
    • L207: dominant frame-level representation
    • L209: static ambiguity
  • Minor

    • L213, L220: Eq. equation 1 →\rightarrow Eq. (1)
Reviewer_ZCrfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents TTT4D, a framework that extends ZipMap to dynamic scenes without additional training. The idea presented in the paper is that dynamic regions can be detected as inconsistencies in ZipMap's internal representations and adaptation mechanism. TTT4D combines different cues to reliably differentiate between dynamic regions and newly unobserved geometry (that also creates representation inconsistencies) to create dynamic objects masks. The predicted masks are then used to reduce the influence of the dynamic objects on the reconstructed geometry. The keys and values corresponding to dynamic regions are removed from patch attention in earlier layers; later the keys are restored but their values are replaced with the average value of the static regions.

Critical strengths

The paper presents a training-free method for reconstructing dynamic scenes using predicted motion cues.

The method uses ZipMap's own inconsistencies without needing explicit training data for dynamic scenes.

TTT4D inherits the linear time complexity from ZipMap, significantly reducing the inference time compared to quadratic-time baselines.

Critical weaknesses

It is unclear whether this method can model dynamic regions or just detect them. Since this is the core claim: "saliency vs. motion", it needs convincing evidence. The paper evaluates only on video object segmentation, camera pose estimation and whole-scene 4D reconstruction. The mechanism presented to handle the dynamic regions could simply encourage the model to ignore them, and this is partially supported by Figure 6 in the paper. Please provide an evaluation restricted to the dynamic regions to understand this effect better.

On 4D evaluation (DyCheck), TTT4D performs worse than VGGT, which has no mechanism to handle dynamic scenes, and the efficiency gains claimed in the paper can largely be attributed to the ZipMap. Additionally, several recent feed-forward methods for handling dynamic scenes like PI3, VGGT-Omega, DPM, Geo4D, Align3r, Uni4D, St4RTrack, StreamVGGT, STream3R, Point3R, ... are not provided as comparisons or discussed appropriately in the related work section.

Questions for authors

Can the authors provide a clear evaluation of the reconstruction of dynamic objects compared to ZipMap and recent methods as listed in the weaknesses section?

Can the authors provide additional visualizations of the reconstructed areas for dynamic objects specifically?

Reviewer_F4ajOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper presents TTT4D, a training-free framework that adapts the pretrained ZipMap model to dynamic 4D reconstruction. The method extracts four internal cues—write-aware adaptation demand, frame-centered query deviation, gate-update magnitude, and value-channel dispersion—and combines them to estimate dynamic regions. The resulting masks are refined using hysteresis thresholding and GrabCut, then used to guide shallow hard-to-soft attention routing. Experiments evaluate dynamic segmentation, camera pose estimation, dense reconstruction, and computational efficiency. The results show improvements over the underlying ZipMap backbone while retaining its linear-time scaling with sequence length.

Critical strengths

Strengths

S1. Relevant problem and practical motivation.

Improving dynamic reconstruction without additional fine-tuning or external motion-estimation models is a worthwhile objective. Building on a scalable stateful backbone makes the approach potentially useful for longer video sequences.

S2. Consistent improvements over the underlying backbone.

Tables 2–3 report improvements over ZipMap across the evaluated pose and reconstruction metrics. For example, the ATE on TUM-Dynamics decreases from 0.014 to 0.009. The dynamic segmentation results also compare favorably with the evaluated training-free alternatives, particularly on DAVIS-2017 and DAVIS-all.

S3. Useful ablations and end-to-end efficiency measurements.

The paper includes both single-cue and leave-one-out evaluations, as well as mask-refinement and layer-range ablations. The efficiency analysis reports cue extraction, mask post-processing, and the second forward pass separately, providing a more informative account than backbone inference time alone.

Critical weaknesses

Weaknesses

W1. The mechanistic justification for the write-aware adaptation score remains insufficient. Equation (1) uses a negative inner-product objective, whereas Eq. (2) introduces the residual term ∥v−fW(k)∥2\|v-f_W(k)\|_2. This residual-based form does not follow directly from the stated objective and the update rule of the nonlinear fast-weight model. I recognize that the authors present it as a proxy rather than an exact gradient norm. Nevertheless, it remains unclear why this particular formulation reliably measures write-time adaptation demand, rather than feature mismatch alone. Since extracting motion evidence from adaptation dynamics is a central claim, clarifying the assumptions, mechanistic connection, or empirical justification for this score would strengthen the interpretation.

W2. The organization of the cues and their fusion could be explained more clearly. The Introduction distinguishes adaptation-level and representation-level evidence, but their correspondence to the four cues, as well as the distinction between primary and auxiliary cues, is scattered across different paragraphs. Explicitly organizing these two classifications at the beginning of the method section would help readers follow the design. Additionally, given the different roles of the four cues, a brief explanation of why they are combined by direct summation after normalization would make the overall rationale clearer.

W3. The incremental contributions of the adaptation-level cues and hard-to-soft routing are not fully isolated. The paper demonstrates improvements over ZipMap, and Tables 5–6 support the benefits of cue fusion for segmentation. However, these results do not fully establish how much the adaptation-level cues improve downstream pose estimation and reconstruction beyond representation-level cues alone. Similarly, Table 8 compares isolation over different depth ranges, rather than directly comparing hard-only and hard-to-soft routing using identical dynamic masks and the same intervened layers. Consequently, the benefits of the soft stage are difficult to distinguish from those of intervening in fewer layers. Since the linear-time backbone is inherited from ZipMap and shallow dynamic masking has precedents, these targeted controls are particularly important for establishing the value of the proposed additions.

Questions for authors

Please see W1–W3 in the Weaknesses section.

Reviewer_QpSSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces TTT4D, a framework for dynamic scene reconstruction built upon the pre-trained ZipMap model, with no extra offline training required. It leverages motion cues derived from fast-weight adaptation and internal features to predict dynamic masks. These masks drive a shallow hard-to-soft attention routing mechanism to suppress interference from dynamic objects. Evaluations on dynamic segmentation, camera pose estimation and dense reconstruction demonstrate consistent gains over ZipMap, while preserving the backbone’s good scalability.

Critical strengths
  1. This work leverages TTT to investigate fast-weight adaptation signals for motion discovery, presenting an alternative paradigm to attention-centric motion modeling.
  2. It enables efficient adaptation to dynamic scenes: the proposed approach augments a pre-trained ZipMap backbone without extra offline training or external motion priors, while retaining its attractive scaling property with respect to sequence length.
  3. The method is validated through extensive empirical evaluations spanning dynamic segmentation, camera pose estimation, and dense reconstruction, yielding consistent gains over the vanilla ZipMap. Ablations on cues and routing mechanisms further verify the contribution of the introduced modules.
Critical weaknesses
  1. Incomplete ablation of motion cues. The current ablation suite lacks the u+ru+r baseline alongside the full model. Without this variant, the complementarity between the two primary cues cannot be clearly demonstrated.
  2. Missing comparison against VGGT-Ω. VGGT-Ω is also capable of dynamic-scene modeling. The manuscript does not clarify the task-level differences nor justify its exclusion from the benchmark. If its code and weights were available before submission, a quantitative comparison on shared pose and geometry metrics would strengthen the evaluation.
Questions for authors
  1. Undifferentiated reconstruction evaluation for static and dynamic regions. Reconstruction accuracy is not reported separately for static and dynamic regions, and cross-frame geometric consistency of static regions is absent. As a result, it is ambiguous whether the performance gains arise from improved geometry estimation of dynamic objects or better reconstruction of the static background.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.