acceptodds
Under review as a conference paper at ICLR 2027

Scr4tch: Self-Supervised 4D Reconstruction from Raw Video Alone

Abstract

Self-supervised learning has enabled powerful visual representations at scale, yet its potential for understanding the 3D structure and dynamics of the world directly from raw video remains largely unrealized. We present Scr4tch, a model that learns geometrically grounded 4D representations entirely from scratch using uncalibrated monocular videos. Without requiring camera calibration, geometric annotations, or pretrained geometric models, Scr4tch unlocks self-supervised learning for 4D understanding from large-scale Internet videos. Our key idea is to learn 4D representations that encode cross-view relationships through latent reconstruction, while constraining how motion is modeled to separate camera motion from scene dynamics and suppress appearance shortcuts. Joint geometric optimization then grounds these representations in an explicit, deformable 4D scene. To train Scr4tch, we additionally collect 2,154 hours of challenging, casually captured videos from YouTube without offline geometric preprocessing. Experiments demonstrate accurate camera estimation, 4D reconstruction, and high-quality novel view synthesis on unseen videos, establishing a path toward scalable self-supervised learning of the dynamic world.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
4D reconstruction, Self-supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_nSRpOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Scr4tch addresses the problem of learning a generalizable, explicitly renderable 4D scene representation from uncalibrated monocular RGB videos, without camera annotations, geometric supervision, or pretrained geometric models. The method first learns latent reconstruction using a shared transformer that predicts camera, time, and motion tokens. Cross-split camera-token exchange, appearance augmentation, and nested dropout on motion tokens are intended to discourage reconstruction shortcuts and separate viewpoint changes from scene dynamics. A second training stage introduces explicit cameras and deformable Gaussians, together with motion masks, camera gradient routing, and self-supervised depth consistency losses. The paper also introduces Casual YouTube, a collection of 2,154 hours of video, and evaluates camera estimation, novel-view synthesis, and geometric and motion representation probing.

Critical strengths
  • The paper tackles a meaningful extension of self-supervised reconstruction: preserving and reconstructing dynamic content while learning cameras, rather than treating dynamics solely as a nuisance. The analysis distinguishes several shortcut pathways—through camera, motion, and time conditioning—and assigns different constraints to them. The context-only Gaussian pass and the shared explicit camera conditioning are particularly sensible design choices. Although several components have clear precedents, their integration into a system producing explicit dynamic reconstructions under these training assumptions is potentially significant.

  • The evaluation covers complementary capabilities: static and dynamic camera estimation, static and dynamic novel-view synthesis, depth probing, and motion segmentation probing. The camera results are strong among the compared self-supervised methods: on Wild-SLAM, RTA@5 is 0.528, compared with 0.319 for E-RayZer, and RRA@5 reaches 1.000 in Table 1. The paper includes meaningful component ablations and explicitly excludes held-out target images from NVS inputs. Reporting only explicit Gaussian rendering for dynamic NVS, while acknowledging possible leakage in latent rendering, is a useful methodological precaution.

  • The motivation is clear, and the progression from latent reconstruction to explicit alignment is understandable. Figures 2, 3, and 5 illustrate concrete failure modes rather than only displaying favorable reconstructions. The appendix adds useful explanations of gradient routing, Gaussian deformation, and depth supervision, and appropriately distinguishes raw self-supervised masks from masks obtained through supervised probing.

  • Learning geometric representations from ordinary video without offline geometric preprocessing is relevant to scalable visual learning. The representation results are encouraging: the motion probe achieves 0.766 IoU, compared with 0.665 for DINOv2, and multi-view depth probing achieves 0.056 ARE in Table 3. These results suggest that the learned backbone contains useful information beyond what is needed for image reconstruction. The Casual YouTube ablation also provides evidence that less curated video can improve the system.

Critical weaknesses

W1. The specific benefit of nested motion-token compression is not fully isolated.

Table 5 shows that removing nested dropout substantially degrades camera estimation, supporting the importance of constraining motion conditioning. However, this ablation simultaneously changes the average token budget, prefix ordering, and exposure to absent motion/time conditioning during pretraining. It therefore leaves unclear which aspect produces the improvement.

A comparison with a small fixed motion-token budget and binary dynamic-state dropout under matched training conditions would clarify the methodological contribution. The existing results support the overall recipe, while these controls would strengthen the proposed compression-based explanation.

W2. The evaluation provides limited direct evidence for the accuracy of moving geometry.

Dynamic novel-view synthesis and the camera–time matrices provide encouraging evidence of controllable reconstruction. However, full-image rendering metrics can conceal errors around moving objects, particularly when static backgrounds occupy much of the image. Meanwhile, Tables 3 and 4 evaluate supervised probes on frozen features, rather than the original depths and motion masks produced by the reconstruction model.

Foreground-specific NVS metrics and a direct quantitative assessment of the native dynamic geometry would make the conclusions more convincing. The temporal scope should also be stated precisely: the current dynamic NVS protocol evaluates novel viewpoints at observed input times.

W3. Several reporting and implementation details require clarification.

The final cumulative configuration in Table 5 differs from the main results without an explanation of the experimental settings. For example, Wild-SLAM RRA@5 is 0.906 in Table 5 versus 1.000 in Table 1. The ablation labels also omit LdistillL_{\mathrm{distill}} and LregL_{\mathrm{reg}}, leaving their inclusion unclear. These differences may reflect separate training configurations, but the relationship between the experiments should be explicit.

Additionally, Eq. (16) directly compares single-view and multiview log depths despite the acknowledged monocular scale ambiguity. The authors should clarify whether normalization, alignment, or a shared learned scale convention makes these predictions compatible.

Questions for authors
  1. Why does the final cumulative configuration in Table 5 differ from Tables 1–2, particularly Wild-SLAM RRA@5 of 0.906 versus 1.000? Which losses and training settings are shared?

  2. How does nested dropout compare with a small fixed motion-token budget or binary dynamic-state dropout under matched conditions?

  3. Can you provide foreground-specific NVS results or a direct quantitative assessment of the native moving geometry?

  4. How are depth scales reconciled in Eq. (16), and how are time tokens obtained for timestamps without an observed image?

6 comments

Dear Reviewer nSRp,

Thank you for your careful review and for recognizing our design choices, in particular the context-only Gaussian pass and the shared explicit camera conditioning. We are glad that the failure-mode visualizations helped convey our motivation. Your suggestions are very helpful, and we are revising the paper accordingly. We apologize for the late reply, as we wanted to run the additional experiments needed to answer your questions as thoroughly as possible. Our responses are below.

W1 & Q2. Isolating the benefit of nested motion-token compression

Thank you for suggesting these two very constructive controls. Binary dynamic-state dropout is similar to the strategy used in RayDer. We trained lightweight models for the three variants under identical settings, using 4 motion tokens for the small fixed budget, and probed the learned features. Together, these controls separate the three factors you identified: the token budget, the exposure to absent motion conditioning, and the ordered (prefix) compression.

Camera probe:

VariantAUC@30 ↑RRA@30 ↑RTA@30 ↑
Small fixed budget (4 tokens)0.2500.9260.373
Binary dropout0.2800.9400.406
Nested dropout0.2920.9470.423

Flow probe on Spring:

VariantEPE ↓Fl-all ↓
Small fixed budget (4 tokens)3.470.292
Binary dropout2.690.196
Nested dropout2.290.148

Token budget and absent motion conditioning (cameras). The small fixed budget uses only 4 motion tokens, yet it gives clearly the worst camera features. A small budget alone therefore does not remove the shortcut. Because the motion tokens are always available, they always provide a path that bypasses the camera and encodes the target image directly, however small the budget. Binary and nested dropout both close this path by regularly dropping the motion pathway, which forces the camera pathway to explain viewpoint change. As a result, both learn clearly better camera features.

Ordered compression (motion). Binary and nested dropout both expose the model to absent motion conditioning, but they differ when the motion tokens are present. Binary dropout leaves them uncompressed and unordered. Nested dropout instead compresses information toward the leading tokens, which further limits target-appearance leakage and promotes correspondence learning. This is reflected in the flow probe, where nested dropout is clearly best (EPE 2.29 vs. 2.69). In the style-bootstrap experiment of Fig. 2, the binary-dropout variant also fails to predict context-consistent colors, and we will add this visualization to the revision.

Finally, different dynamic scenes need different numbers of motion degrees of freedom. Nested dropout does not commit to a fixed budget in advance. It only requires the motion information to be as compressed as possible, which a fixed budget cannot provide.

W2 & Q3. Direct evidence for moving geometry

We agree that full-image metrics can hide errors on moving objects. We add foreground-specific NVS metrics and directly evaluate the native depth and motion masks produced by the model.

Foreground NVS. We evaluate on two additional datasets, DyCheck and DyNeRF (9 input frames, one camera per timestep), and report foreground metrics:

Each cell: DyCheck / DyNeRF.

MethodSelf-sup.PSNR ↑SSIM ↑LPIPS ↓PSNR-fg ↑SSIM-fg ↑LPIPS-fg ↓
NeoVerse✗8.77 / 15.780.167 / 0.4320.586 / 0.3569.98 / 19.300.170 / 0.4250.560 / 0.334
C4G✗12.93 / 18.950.298 / 0.4690.631 / 0.35611.00 / 19.170.217 / 0.4580.664 / 0.394
MoVieS✗12.33 / 24.890.267 / 0.8400.478 / 0.12711.78 / 23.400.207 / 0.6930.532 / 0.181
Ours✓13.54 / 20.540.265 / 0.5230.546 / 0.25111.92 / 21.090.204 / 0.5670.580 / 0.243

We also report foreground metrics on NVIDIA Dynamic (Table 2 setting):

MethodSelf-sup.PSNR-fg ↑SSIM-fg ↑LPIPS-fg ↓
MoVieS✗15.310.2830.433
C4G✗14.100.2020.589
NeoVerse✗15.290.2790.363
Ours✓14.930.2270.470

Ours is the only self-supervised method here, while all baselines rely on geometric supervision or pretrained geometric models. Even so, Ours achieves the best full-image and foreground PSNR on DyCheck, ranks second on DyNeRF, and outperforms C4G on all foreground metrics on NVIDIA Dynamic. These results show that our model is competitive with supervised methods on the moving regions, and we will add them to the revision.

Native predicted depth. We evaluate the raw depth output of the model (no probe) on TUM-Dynamic, over the full image and on the human foreground:

MethodSelf-sup.AbsRel ↓δ1 ↑AbsRel-fg ↓δ1-fg ↑
MonST3R✗0.2090.7540.2330.693
NeoVerse✗0.1140.8850.1230.892
E-RayZer✓0.2920.5670.3800.457
Ours✓0.2060.6730.2650.569

Ours clearly outperforms the self-supervised E-RayZer and is on par with the supervised MonST3R in full-image AbsRel (0.206 vs. 0.209). NeoVerse and MonST3R achieve stronger foreground results, but both are supervised and built on large pretrained geometric foundation models, which greatly reduce reconstruction ambiguity. Without such supervision, 4D reconstruction from monocular video faces two coupled ambiguities: camera motion vs. scene motion, and the depth of a moving object vs. its motion. Our work jointly learns cameras and renderable dynamic geometry from uncalibrated monocular video, without geometric supervision or pretrained geometric models. Separating camera motion from scene dynamics is a central contribution, while our depth-consistency objectives, including L_sv, are designed to mitigate the depth–motion ambiguity in moving regions. The native-depth results show improvements over the self-supervised E-RayZer in both full-image and foreground metrics, while revealing a remaining gap to supervised methods in foreground geometry. Improving moving-object geometry is therefore an important next step within our self-supervised 4D reconstruction framework.

Native motion masks. We also evaluate the raw self-supervised masks produced by the model (no probe) on Kubric-4D, which provides per-object dynamic labels:

MethodIoU ↑Precision ↑Recall ↑F1 ↑FPR ↓
MonST3R0.0070.0080.0560.0140.265
Ours w/o asymmetric cost0.1930.2170.6060.3030.169
Ours0.2870.3880.5280.4180.069

Since these masks come directly from the reconstruction model, they complement the probing results in Table 4. Self-supervised feed-forward baselines such as E-RayZer do not produce motion masks, so we use MonST3R as the reference. Its masks often include static structure such as the ground, which explains its very low precision. The asymmetric cost targets exactly this failure mode. By favoring static explanations, it lowers the false-positive rate from 0.169 to 0.069 and raises precision from 0.217 to 0.388, with a moderate drop in recall. We will include qualitative mask comparisons in the revision.

Temporal scope. This is a fair point. Our model reconstructs one scene state per input frame, in the same per-timestep manner as MonST3R. The dynamic NVS protocol therefore evaluates novel viewpoints only at the observed timestamps, and we will state this scope explicitly in the paper.

W3 & Q1. Table 5 vs. Tables 1–2, and the included losses

We apologize for the missing description. The full-model row of Table 5 is the same model as in Tables 1–2, and the discrepancy comes from the evaluation protocol. Tables 1–2 must use settings that all baselines support. Table 5 compares only our own variants and therefore uses the protocol matching our training setting:

  • Camera. Table 5 runs non-overlapping 9-frame windows and computes RRA/RTA over all pairs within each window. Table 1 uses windows that share boundary frames and reports adjacent-pair metrics, which all baselines support. The different windowing also changes the intrinsics estimate and hence AREf.
  • NVS. Table 5 uses 9 input frames, our training length, so that the results are not affected by running on inputs longer than those seen in training. Table 2 uses 12 frames to match the baselines.

All rows of Table 5 share the same protocol, so the relative comparisons hold. In the revision, we will describe this protocol of Table 5 and explain how its numbers relate to those in Tables 1–2.

Regarding the losses: L_distill, L_reg and the context-only Gaussian pass came from early experiments and are always enabled in Stage 2. In Table 5, they are included in the L_exp row and in every row below it. We apologize for omitting them from the labels and will fix this.

Q4 (a). Depth scale in Eq. (16). We apply no explicit normalization or alignment. Instead, the model is encouraged to adopt a shared learned scale convention for a given scene across different inference passes. Two mechanisms enforce this:

  1. Eq. (16) directly penalizes any scale difference between single-view and multi-view log-depths, so the model is pushed to predict the same scale in both settings.
  2. As described in Appendix C.1, the context-only pass predicts Gaussians from the initial (not bootstrapped) camera tokens to prevent camera leakage. These Gaussians are then unprojected with the cameras predicted by the bootstrap pass. This requires the depth scale of the context-only pass to agree with the translation scale of the bootstrap cameras.

Q4 (b). Time tokens for unobserved timestamps. As noted under W2, our current formulation reconstructs the scene state at observed timestamps. Each time token is predicted from the image observed at that timestamp, so we do not obtain time tokens for unobserved times, and temporal interpolation is not constrained during training. Since the time token has a single degree of freedom, interpolating between neighboring time tokens is a natural extension and an interesting direction for future work.

In addition, we would like to share a related result. Although temporal interpolation is not trained, the backbone already learns dense cross-time correspondence, which we believe is a useful basis for such an extension. A frozen-backbone optical-flow probe with probe heads of comparable size shows this:

Each cell: Sintel / Spring.

BackboneEPE ↓Fl-all ↓
CroCo v223.62 / 4.050.815 / 0.285
E-RayZer24.24 / 5.720.814 / 0.449
VideoMAEv217.72 / 2.730.543 / 0.179
Ours14.48 / 2.580.570 / 0.178

Ours achieves the lowest EPE on both Sintel and Spring and the best Fl-all on Spring, and is behind VideoMAEv2 only in Sintel Fl-all. It also clearly outperforms CroCo v2 and E-RayZer. This suggests that our features capture not only viewpoint changes but also how scene content moves over time.

We thank you again for your valuable feedback and hope our responses address your concerns. We are incorporating these results into the revised manuscript and would be happy to discuss any further questions.

Reviewer_F2bDOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Scr4tch, a new self-supervised 4D reconstruction model that does not rely on any external annotations. Scr4tch takes a monocular video, and predict cameras and per-view Gaussians. Scr4tch adopts a two-stage training strategy with novel-view synthesis as the major training objective. The first stage is a latent-based pre-training stage that learns strong priors about scene and motion. The second stage aligns explicit Gaussian prediction to the learned latent priors, which enables geometrically grounded predictions. To prevent appearance-based shortcut solutions, a few techniques are proposed, e.g., motion token nested dropout, constrained time token dimension, swapped camera token between the two splits. To train this system, 2,154 hours of YouTube videos are collected, mixed with existing video datasets.

Extensive evaluations show that: Scr4tch achieves strong results on novel-view synthesis and camera pose prediction, across static and dynamic benchmarks. Also, Scr4tch learns strong scene and motion priors, probed via depth estimation and motion segmentation.

Critical strengths

(1) First of all, this paper is trying to address an important but challenging topic: self-supervised learning from monocular videos, which involves the understanding of 3D scenes and motions. This is also very relevant in the context of "world modeling", which requires the understanding of viewpoint and action.

(2) The proposed method, Scr4tch, is well-designed, and each component is well justified.

(3) Empirically, Scr4tch achieves strong results compared to supervised and self-supervised methods on novel-view synthesis and pose estimation. Scr4tch also learns strong priors about depth and motion, compared with commonly used backbones, demonstrated via probing results.

(4) I found some visuals to be interesting and compelling (e.g., Fig. 2).

Critical weaknesses

(1) Insufficient evaluation. Currently, Scr4tch is only evaluated on Wild-SLAM Mocap in terms of dynamic reconstruction. The results on more dynamic benchmarks, e.g., DyCheck, Sintel, and TUM-Dynamic, would be helpful. Also, I am wondering its comparison with"real state-of-the-art" 3D/4D reconstruction models, e.g., VGGT-Omega, DA3. It's OK that the proposed method cannot beat these potentially larger and stronger supervised systems, but I'm curious the gap, and if the proposed method and its follow-ups can close the gap in the future. Moreover, it's a bit surprising to be that some methods that trained only with static scenes (e.g., E-RayZer) can do fairly well on Wild-SLAM Mocap, so I wonder if this benchmark is not "dynamic enough" as a benchmark for dynamic reconstruction.

(2) Insufficient analysis on data and scaling. The promise of self-supervised learning lies in scaling things up with unannotated data. However, I don't see analysis on model scaling (model size, training data, iteration), which poses some concerns about whether the proposed system can scale up.

Also, the understanding of data mixture also matters but is missing from the current submission. What's the principle for current data mixing strategy? For example, what's the ration between static scenes and dynamic scenes? Also, we know that static scenes are easier to train, what's the principle for selecting dynamic data? Can the model still work with scenes with significant camera and scene motion? How does the choice of data affect downstream tasks?

(3) The proposed method adopts a two-stage training strategy, i.e., first latent then explicit reconstruction. The effect of two-stage training is missing from the submission. Does the proposed method still work without stage 1? How much does stage 1 training affect performance? Does training longer help? Given a fixed compute budget, how to allocate it between two stages? It's OK that if answering all questions is hard, but I'd like to have a sense about how and why two-stage training works.

Questions for authors

Minor Concerns and Questions:

(1) Why can't we exchange time embeddings for two splits as well (as what we did for camera tokens, they are all shared across two splits)?

(2) More qualitative results are needed in the submission (yet, it's understandable that the page number for main paper is limited).

(3) Can we have some quantitative results in terms of the accuracy of the learned raw motion masks (not the probing results)? For example, can we report precision and recall (or other reasonable metrics) for the motion masks, and compare with other self-supervised systems that learn motion masks?

(4) Some ablation studies (they are not as important as what I mentioned in "Critical Weaknesses") are missing. For example, how does the choice of augmentation strategies affect training. It's important to understand what kind of augmentations that the concept of motion is invariant to. Some quantitative comparisons for the choice of dimension of motion token, time token, etc, would also be useful for understanding the proposed system. Overall, I'm wondering if the proposed work can answer a more general question of what's the principle to disentangle camera motion and scene motion, which should be very useful for even other domains.

7 comments

Dear Reviewer F2bD,

Thank you for the detailed review and the positive assessment of our method. We are excited that you found visuals such as Fig. 2 as compelling as we did. Your comments are very helpful, and we are revising the paper accordingly. We apologize for the late reply, as we wanted to finish the additional experiments and answer your questions as completely as possible. We address each point below.

W1. Insufficient evaluation

(a) More dynamic benchmarks and a state-of-the-art supervised reference. As suggested, we evaluate on TUM-Dynamic, DyCheck and Sintel with full-trajectory metrics. Among recent large-scale supervised reconstruction models such as VGGT-Omega and DA3, we select DA3-Giant-1.1 as the representative reference. DA3, NeoVerse and MonST3R process each sampled sequence in a single pass. Ours and E-RayZer run sliding windows chained through overlapping frames, with no GT used for stitching. ATE/RPE use one global Sim(3) alignment, and drift is the end-point error / path length after calibrating on the first 5 frames only.

Each cell: TUM / DyCheck / Sintel.

MethodSelf-sup.RRA@5 ↑RTA@5 ↑RTA@15 ↑RPE-t ↓RPE-r (°) ↓ATE ↓Drift ↓
DA3-Giant-1.1✗1.000 / 1.000 / 0.9970.278 / 0.646 / 0.3680.549 / 0.894 / 0.6390.0164 / 0.0062 / 0.1050.531 / 0.308 / 0.8690.0205 / 0.0068 / 0.2340.0194 / 0.0216 / 0.220
NeoVerse✗0.965 / 0.998 / 0.9840.169 / 0.243 / 0.1590.469 / 0.612 / 0.4920.0825 / 0.0253 / 0.1567.54 / 1.09 / 1.390.0983 / 0.0226 / 0.2960.0192 / 0.0638 / 0.378
MonST3R✗0.631 / 0.617 / 0.9380.024 / 0.059 / 0.2430.084 / 0.229 / 0.4270.0726 / 0.0547 / 0.1357.74 / 6.64 / 2.420.275 / 0.112 / 0.1860.191 / 0.318 / 0.960
E-RayZer✓0.981 / 0.917 / 0.7880.143 / 0.170 / 0.0940.427 / 0.477 / 0.2430.0318 / 0.0392 / 0.1691.52 / 2.87 / 6.280.127 / 0.0626 / 0.5250.0645 / 0.109 / 0.655
Ours✓0.990 / 0.975 / 0.9600.114 / 0.177 / 0.1180.403 / 0.565 / 0.4210.0292 / 0.0249 / 0.1091.29 / 1.82 / 2.190.111 / 0.0442 / 0.2560.0554 / 0.0733 / 0.562

Ours outperforms E-RayZer on 22 of the 24 entries and MonST3R on 21 of 24, while it is behind on some translation metrics (TUM RTA vs. E-RayZer, and Sintel RTA/ATE vs. MonST3R). As expected, DA3 leads on nearly all metrics. It is substantially larger, trained with dense geometric supervision on large labeled 3D datasets, and uses far more compute.

(b) Is Wild-SLAM Mocap "dynamic enough"? This is a good observation, and we agree that Wild-SLAM is dominated by camera motion with moderate scene dynamics. The new results support this. As dynamics increase, the gap between Ours and the static-scene E-RayZer becomes much larger. On Sintel, RRA@5 is 0.960 vs. 0.788, RTA@15 is 0.421 vs. 0.243, and ATE is 0.256 vs. 0.525. On crowded scenes (see W2 below), RTA@5 is 0.769 vs. 0.256. E-RayZer's strong Wild-SLAM result thus mainly reflects the benchmark's limited dynamics. We will add these benchmarks to the paper.

(c) How large is the gap, and can it be closed? One advantage of a purely self-supervised objective is that it can be optimized at test time on the input video alone. We apply test-time training (TTT) on Wild-SLAM Mocap using only our self-supervised loss on the test input frames. No GT is used for training or checkpoint selection.

Wild-SLAM Mocap with test-time training.

MethodResolutionAREf ↓RRA@5 ↑RRA@15 ↑RRA@30 ↑RTA@5 ↑RTA@15 ↑RTA@30 ↑
Ours256×2560.0401.0001.0001.0000.5280.9300.992
Ours w/ TTT256×2560.0331.0001.0001.0000.6540.9650.995
NeoVerse336×5600.0200.9871.0001.0000.7050.9360.987
DA3-Giant-1.1504×5040.0150.9921.0001.0000.8870.9790.987

TTT keeps the rotation accuracy and raises RTA@5 from 0.528 to 0.654, narrowing the gap to the supervised NeoVerse. It outperforms NeoVerse on RTA@15 and achieves the best RTA@30, despite a much lower input resolution. Together with the scaling trends below, this suggests that the gap to supervised systems is not a hard limit, and that follow-ups can narrow it further with more capacity, data, resolution and test-time adaptation.

W2. Data and scaling

Model size and training time. Due to limited compute and time, a full scaling sweep was not feasible, but our experiment records already show clear trends. For Stage 1, we measure latent NVS with the target-aware protocol of RayZer:

Stage 1PSNR ↑SSIM ↑LPIPS ↓#Params
Ours-1216.900.4820.288345.6M
Ours-2419.910.6590.175645.5M
Training time:24h48h72h
Stage 1 (Ours-24), PSNR ↑17.8818.6319.91
Stage 2, AREf ↓0.0550.0440.040
Stage 2, RRA@5 ↑0.8261.0001.000
Stage 2, RTA@5 ↑0.4290.4910.528

Both Stage-1 models were trained for the same time. Ours-12 stopped improving after about 24h, whereas Ours-24 kept improving until 72h. The larger model therefore both reaches a higher level and makes better use of additional compute. Stage 2 converges faster than Stage 1, which we discuss under W3.

Training data. Besides the dynamic-scene ablation in the paper, we ran the "w/o Casual YouTube" ablation on the static DL3DV benchmark:

MethodAREf ↓RRA@5 ↑RTA@5 ↑PSNR ↑
Ours w/o Casual YouTube0.1980.9210.76421.65
Ours0.1340.9310.78921.70

Removing YouTube increases the share of static training data, yet the model still performs worse on static scenes. More unlabeled in-the-wild video therefore benefits both dynamic and static reconstruction. The largest gain is in focal estimation, which we explain below.

Data mixture principle. As stated in Sec. 4.1, the four training datasets are sampled at a 5:3:1:1 ratio. The principle is to let Casual YouTube, our unlabeled and most challenging source, dominate the mixture, as a hard test for self-supervised training. The resulting dynamic-to-static sampling ratio is about 8:2. It is not exact, because "dynamic" datasets also contain some static scenes.

Selecting dynamic data. The only manual step is at download time, and selection is done per channel rather than per video. We search for POV videos with keywords such as walk and cycle. We keep channels whose footage is real-world, reasonably sharp, and not always covered by large moving objects (e.g., in-car driving views). We intentionally do not filter by weather or pedestrian density, to improve generalization to complex scenes. Videos within a selected channel are not filtered further.

After download, there is no offline preprocessing. While streaming videos in the dataloader, we only split clips at (i) scene cuts, which break multi-view consistency, and (ii) completely static segments, which carry no reconstruction signal. Both are detected with a simple threshold on the frame-to-frame mean absolute difference. We keep this rule simple on purpose, so some samples still contain transitions, slides or text overlays. Even so, Casual YouTube substantially improves performance, which shows that our self-supervised training is robust to noisy in-the-wild data.

Effect of data on downstream tasks. The largest gain is in intrinsics, with AREf improving from 0.198 to 0.134 on DL3DV. Many existing datasets are large but captured with only a few devices, so they offer little focal-length diversity. Casual YouTube is much more diverse: it covers many cameras and focal lengths, sometimes even with zoom changes within a single video. This allows the model to learn much richer intrinsics information.

Significant scene motion. We select walking-in-crowd sequences in which potentially dynamic objects, estimated with SAM3, cover on average 39.3% of the image. Since GT cameras are unavailable, we use DA3-Giant-1.1 predictions as a pseudo-reference:

MethodRRA@5 ↑RTA@5 ↑AUC@20 ↑
MonST3R1.0000.2690.409
NeoVerse1.0000.8590.846
E-RayZer0.9740.2560.528
Ours1.0000.7690.836

Even with such large moving foregrounds, our camera estimates remain robust and are close to NeoVerse. We will visualize these cases in the revision.

Significant camera motion. We group ScanNet pairs by view overlap: High (H) is 50–90% and Low (L) is 20–50%. Since the H group contains fewer pairs, several methods obtain identical scores.

MethodRRA-H@15 ↑RTA-H@15 ↑AUC-H@20 ↑RRA-L@15 ↑RTA-L@15 ↑AUC-L@20 ↑
MonST3R0.9380.5000.3490.3500.0500.035
NeoVerse1.0000.9380.7200.9250.6750.439
E-RayZer0.9380.2810.2080.3250.0750.041
Ours0.9380.8750.5820.8500.1250.080

Ours is on par with or better than E-RayZer and MonST3R in both groups, and is second only to the supervised NeoVerse at high overlap. At low overlap, NeoVerse is stronger in translation, which is expected. Self-supervised photometric objectives rely on sufficient overlap among the inputs, while supervised methods can be trained directly on low-overlap pairs sampled from labeled datasets. Extending self-supervision to sparse, low-overlap inputs, e.g., through pose self-distillation, is an interesting direction for future work.

W3. Why two-stage training?

Two-stage training is essential. Without Stage 1, the model does not converge to a meaningful solution.

MethodAbsRel (TUM-Dyn.) ↓RRA@5 (Wild-SLAM) ↑RTA@5 (Wild-SLAM) ↑
w/o Stage 10.2890.3050.019
w/o Stage 2N/A0.5120.011
Full0.2061.0000.528
  • Without Stage 1, the explicit outputs (cameras, deformable Gaussians and motion masks) must be learned jointly from scratch. This makes optimization very hard, and translation estimation fails completely.
  • Without Stage 2, the model learns latent reconstruction, but the latent space is not aligned with Euclidean geometry, so explicit cameras are inaccurate.

Stage 1 thus learns the reconstruction and correspondence features under an easier latent objective, and Stage 2 mainly aligns them with an explicit Euclidean representation. In addition, the latent static and dynamic renderings learned in Stage 1 provide the initial signal for learning motion masks.

Longer training and compute allocation. Longer training helps in both stages (see W2). In our experiments, the two stages simply received equal training time. Because Stage 2 mostly aligns rather than re-learns, it converges much faster (RRA@5 already reaches 1.000 at 48h). This suggests that, under a fixed budget, shifting compute toward Stage 1 is a good choice.

Reviewer_sucMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Scr4tch, a feed-forward model that learns 4D scene representations from uncalibrated monocular RGB video, using no camera labels, geometric pseudo-labels, or pretrained geometric models. A single VGGT-style alternating-attention transformer is trained in two stages. In latent pretraining, the model limits target-appearance leakage via XFactor-style cross-split exchange of camera tokens, FlexTok-style nested dropout on motion tokens, and a time-token bottleneck with one degree of freedom. Then, the model can decode cameras, soft motion masks, and pixel-aligned deformable Gaussians. Extensive experiments have demonstrated the effectiveness of the proposed method.

Critical strengths

S1

Without camera labels or pretrained geometric networks, the model reaches RRA@5 1.000 and RTA@5 0.528 on dynamic Wild-SLAM Mocap (Table 1). This exceeds every self-supervised baseline (best RTA@5: 0.319, E-RayZer) and the supervised MonST3R (0.357), which runs at higher resolution. On static DL3DV, it is also strong.

The frozen features also give clearly better motion-segmentation probes than DINOv2, CroCo v2, VideoMAE and V-JEPA 2 (IoU 0.766 vs. ≤0.665). This matters most to my assessment: it shows that removing geometric prerequisites need not cost accuracy, which makes the "learn 4D from Internet video" agenda concrete rather than aspirational.

S2

The paper explains why self-supervised dynamic reconstruction collapses. Uncompressed motion tokens carry target appearance and absorb viewpoint change, so the camera pathway degenerates. The paper backs this with diagnostics, such as Fig.2, Fig. 5, Table 5, and Fig.7. These insights should transfer beyond this architecture, and they lift the paper above a system description.

Critical weaknesses
  1. The evaluation of dynamics and 4D is narrow. The title claims 4D reconstruction, but the quantitative evidence for the dynamic part is thin. Dynamic pose is tested on one dataset (Wild-SLAM Mocap) with adjacent-pair relative metrics only. These hide trajectory drift, and the paper reports no ATE/RPE. Dynamic NVS also uses one dataset (NVIDIA Dynamic). The PSNR margin over MoVieS is minor (0.09 dB), while SSIM and LPIPS are worse. Further, nothing evaluates dynamic geometry directly (dynamic depth, scene flow or tracking); the depth probes use static datasets. As a result, the core 4D claim thus rests largely on qualitative figures.

  2. The ablations seem to be incomplete and inconsistent with the main tables. The full-model row of Table 5 does not match the main results, and the paper does not explain why. Table 5 reports RRA@5 0.906, RTA@5 0.592, AREf 0.045 and PSNR 17.87. Tables 1 and 2 report 1.000, 0.528, 0.040 and 17.80 for the same metrics.

  3. Also, several components have only qualitative support or no ablation at all, e.g., the asymmetric cost, cross-split exchange, the time-token bottleneck, appearance augmentation, the context-only Gaussian pass, L_distill, N_m, and two-stage versus single-stage training. The scaling claim rests on one "w/o Casual YouTube" run, confounded with the mixture proportions. For a method with this many components, it is hard to tell which parts are necessary.

Questions for authors
  1. Why does the full-model row in Table 5 differ from Tables 1–2? Was it a different checkpoint, training length, protocol or subset? If the ablations use shorter training, please state this
  2. Are the baselines run from official checkpoints, or retrained on your data mixture?
9 comments

Dear Reviewer sucM,

Thank you for your thorough and constructive review. We are glad that you found the results meaningful for the goal of "learning 4D from Internet video", and that you agreed with our analysis of shortcut-induced collapse (Figs. 2, 5 and 7, and Table 5). Your comments helped us improve the paper a lot. We also apologize for the late reply: we wanted to complete the additional experiments so that we could answer your questions as thoroughly as possible. We address each point below, and all new results will be included in the revision.

W1. Narrow evaluation of dynamics and 4D

Following your suggestion, we extended the dynamic evaluation in five aspects: full-trajectory camera metrics, dynamic NVS with foreground metrics, direct evaluation of the predicted dynamic depth, a correspondence (optical-flow) probe, and direct evaluation of the raw self-supervised motion masks (reported under W3, together with the asymmetric-cost ablation).

(a) Full-trajectory camera evaluation. We evaluate on TUM-Dynamic, DyCheck and Sintel and report RPE, ATE and trajectory drift in addition to RRA/RTA. MonST3R and NeoVerse process each sampled sequence in a single pass. Ours and E-RayZer run sliding windows chained through overlapping frames, and no GT is used for stitching. ATE/RPE use one global Sim(3) alignment per sequence. Drift (end-point error / GT path length) is computed after calibrating on the first 5 frames only, so alignment cannot hide accumulated error.

Each cell: TUM / DyCheck / Sintel.

MethodSelf-sup.RRA@5 ↑RTA@5 ↑RTA@15 ↑RPE-t ↓RPE-r (°) ↓ATE ↓Drift ↓
NeoVerse✗0.965 / 0.998 / 0.9840.169 / 0.243 / 0.1590.469 / 0.612 / 0.4920.0825 / 0.0253 / 0.1567.54 / 1.09 / 1.390.0983 / 0.0226 / 0.2960.0192 / 0.0638 / 0.378
MonST3R✗0.631 / 0.617 / 0.9380.024 / 0.059 / 0.2430.084 / 0.229 / 0.4270.0726 / 0.0547 / 0.1357.74 / 6.64 / 2.420.275 / 0.112 / 0.1860.191 / 0.318 / 0.960
E-RayZer✓0.981 / 0.917 / 0.7880.143 / 0.170 / 0.0940.427 / 0.477 / 0.2430.0318 / 0.0392 / 0.1691.52 / 2.87 / 6.280.127 / 0.0626 / 0.5250.0645 / 0.109 / 0.655
Ours✓0.990 / 0.975 / 0.9600.114 / 0.177 / 0.1180.403 / 0.565 / 0.4210.0292 / 0.0249 / 0.1091.29 / 1.82 / 2.190.111 / 0.0442 / 0.2560.0554 / 0.0733 / 0.562

Ours outperforms the self-supervised E-RayZer on 19 of the 21 entries and the supervised MonST3R on 18 of 21. All remaining cases are translation metrics: TUM RTA vs. E-RayZer, and Sintel RTA/ATE vs. MonST3R. Ours even achieves a lower RPE-t than the supervised NeoVerse on all three datasets, while NeoVerse is stronger in drift and most ATE values. The advantage over self-supervised methods therefore also holds at the trajectory level and is not a side effect of using adjacent-pair metrics.

(b) Dynamic NVS on more datasets, with foreground metrics. We add DyCheck and DyNeRF (9 input frames, one camera per timestep) and also report metrics restricted to the dynamic foreground.

Each cell: DyCheck / DyNeRF.

MethodSelf-sup.PSNR ↑SSIM ↑LPIPS ↓PSNR-fg ↑SSIM-fg ↑LPIPS-fg ↓
NeoVerse✗8.77 / 15.780.167 / 0.4320.586 / 0.3569.98 / 19.300.170 / 0.4250.560 / 0.334
C4G✗12.93 / 18.950.298 / 0.4690.631 / 0.35611.00 / 19.170.217 / 0.4580.664 / 0.394
MoVieS✗12.33 / 24.890.267 / 0.8400.478 / 0.12711.78 / 23.400.207 / 0.6930.532 / 0.181
Ours✓13.54 / 20.540.265 / 0.5230.546 / 0.25111.92 / 21.090.204 / 0.5670.580 / 0.243

Ours is the only self-supervised method in this comparison, while all baselines rely on geometric supervision or pretrained geometric models. Even so, Ours obtains the best full-image and foreground PSNR on DyCheck, and on DyNeRF it ranks second, consistently ahead of C4G and NeoVerse. These results show that our model is competitive with supervised methods on dynamic NVS, and we will add them to the revision.

(c) Direct evaluation of dynamic geometry. We evaluate the raw depth predicted by the model (no probe) on TUM-Dynamic, over the full image and on the human foreground.

Native predicted depth on TUM-Dynamic.

MethodSelf-sup.AbsRel ↓δ1 ↑AbsRel-fg ↓δ1-fg ↑
MonST3R✗0.2090.7540.2330.693
NeoVerse✗0.1140.8850.1230.892
E-RayZer✓0.2920.5670.3800.457
Ours w/o Stage 1✓0.2890.5470.4410.375
Ours✓0.2060.6730.2650.569

Ours clearly outperforms the self-supervised E-RayZer and is on par with the supervised MonST3R in full-image AbsRel (0.206 vs. 0.209). NeoVerse and MonST3R achieve stronger foreground results, but both are supervised and built on large pretrained geometric foundation models. Without such supervision, 4D reconstruction from monocular video faces two coupled ambiguities: camera motion vs. scene motion, and the depth of a moving object vs. its motion. Our work focuses on the first one, which is what allows accurate cameras without any labels. We also reduce the second one with depth-consistency losses such as L_sv. Further resolving the depth–motion ambiguity without supervision is an important and interesting direction for future work. The "w/o Stage 1" row is discussed under W3.

(d) Scene flow and tracking. Like MonST3R, our current formulation reconstructs the scene state at each observed timestamp. It does not explicitly model scene flow or long-range tracks in the explicit (Stage 2) representation, and we will state this scope clearly in the paper. Nevertheless, nested dropout on motion tokens forces the backbone to learn cross-time correspondence (Fig. 2). To quantify this, we train an optical-flow probe on the frozen backbone. The probe heads have comparable sizes: 6.8M parameters for Ours and VideoMAEv2, and 8.7M for CroCo v2 and E-RayZer.

Frozen-backbone flow probe. Each cell: Sintel / Spring.

BackboneEPE ↓Fl-all ↓
CroCo v223.62 / 4.050.815 / 0.285
E-RayZer24.24 / 5.720.814 / 0.449
VideoMAEv217.72 / 2.730.543 / 0.179
Ours14.48 / 2.580.570 / 0.178

Ours achieves the lowest EPE on both datasets and the best Fl-all on Spring, behind VideoMAEv2 only in Sintel Fl-all. This indicates that the learned features encode rich dense motion information. We agree that turning these features into an explicit scene-flow-based 4D Gaussian representation is a promising next step.

W2 & Q1. Table 5 vs. Tables 1–2

We apologize for not stating this clearly. The full-model row of Table 5 is the same model as in Tables 1–2 (same checkpoint and training length). The discrepancy comes entirely from the evaluation protocol. Tables 1–2 compare against many baselines, so their protocol uses only settings that all methods support. Table 5 only compares variants of our own model, so it uses the protocol that matches our training setting (9-frame inputs). This way, the effect of each component is not mixed with the effect of running on inputs longer than those seen in training.

Concretely, for camera estimation, Table 1 uses windows that share boundary frames and reports adjacent-pair RRA/RTA, which all baselines support. Table 5 instead uses non-overlapping 9-frame windows and computes RRA/RTA over all pairs within each window, and this different windowing also slightly changes AREf. For NVS, Table 2 uses 12 input frames to match the baselines, while Table 5 uses 9 frames, our training length.

All rows of Table 5 share the same protocol, so the relative conclusions are valid. In the revision, we will describe this protocol and explain how its numbers relate to those in Tables 1–2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive feedback. We were encouraged that the reviewers found the problem important (F2bD, nSRp), the method well-motivated and the analysis of shortcut-induced collapse insightful, and the camera and representation results strong (all reviewers).

The main shared concern was the scope of the dynamic / 4D evaluation. During the rebuttal we ran the following new experiments, which will be added to the revision:

  1. Full-trajectory camera evaluation on TUM-Dynamic, DyCheck and Sintel, with ATE, RPE and trajectory drift, plus DA3-Giant-1.1 as a state-of-the-art supervised reference (sucM, F2bD).
  2. Dynamic NVS on two more datasets (DyCheck, DyNeRF) and foreground-only NVS metrics, including on NVIDIA Dynamic (sucM, nSRp).
  3. Direct evaluation of the native predicted depth on dynamic scenes, full image and foreground (sucM, nSRp).
  4. Direct evaluation of the raw self-supervised motion masks on Kubric-4D (all reviewers).
  5. A frozen-backbone optical-flow probe on Sintel and Spring (sucM, nSRp).
  6. New ablations: w/o Stage 1, w/o Stage 2, w/o asymmetric cost, nested dropout vs. a small fixed motion-token budget vs. binary dynamic-state dropout, and w/o Casual YouTube on a static benchmark (all reviewers).
  7. Robustness and scaling: crowded scenes with large moving foregrounds, low-overlap camera motion on ScanNet, test-time training, and model-size / training-time trends (F2bD).

We also clarify the evaluation protocol of the ablation table vs. the main comparison tables (sucM, nSRp), which is now stated explicitly in the revised paper. Detailed responses are given below to each reviewer.

We have uploaded a revised manuscript, with all updated content highlighted in blue. We are actively improving the paper further. We thank the reviewers again for their time and valuable feedback, and are always happy to discuss any additional questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.