- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Scr4tch addresses the problem of learning a generalizable, explicitly renderable 4D scene representation from uncalibrated monocular RGB videos, without camera annotations, geometric supervision, or pretrained geometric models. The method first learns latent reconstruction using a shared transformer that predicts camera, time, and motion tokens. Cross-split camera-token exchange, appearance augmentation, and nested dropout on motion tokens are intended to discourage reconstruction shortcuts and separate viewpoint changes from scene dynamics. A second training stage introduces explicit cameras and deformable Gaussians, together with motion masks, camera gradient routing, and self-supervised depth consistency losses. The paper also introduces Casual YouTube, a collection of 2,154 hours of video, and evaluates camera estimation, novel-view synthesis, and geometric and motion representation probing.
Critical strengths
-
The paper tackles a meaningful extension of self-supervised reconstruction: preserving and reconstructing dynamic content while learning cameras, rather than treating dynamics solely as a nuisance. The analysis distinguishes several shortcut pathways—through camera, motion, and time conditioning—and assigns different constraints to them. The context-only Gaussian pass and the shared explicit camera conditioning are particularly sensible design choices. Although several components have clear precedents, their integration into a system producing explicit dynamic reconstructions under these training assumptions is potentially significant.
-
The evaluation covers complementary capabilities: static and dynamic camera estimation, static and dynamic novel-view synthesis, depth probing, and motion segmentation probing. The camera results are strong among the compared self-supervised methods: on Wild-SLAM, RTA@5 is 0.528, compared with 0.319 for E-RayZer, and RRA@5 reaches 1.000 in Table 1. The paper includes meaningful component ablations and explicitly excludes held-out target images from NVS inputs. Reporting only explicit Gaussian rendering for dynamic NVS, while acknowledging possible leakage in latent rendering, is a useful methodological precaution.
-
The motivation is clear, and the progression from latent reconstruction to explicit alignment is understandable. Figures 2, 3, and 5 illustrate concrete failure modes rather than only displaying favorable reconstructions. The appendix adds useful explanations of gradient routing, Gaussian deformation, and depth supervision, and appropriately distinguishes raw self-supervised masks from masks obtained through supervised probing.
-
Learning geometric representations from ordinary video without offline geometric preprocessing is relevant to scalable visual learning. The representation results are encouraging: the motion probe achieves 0.766 IoU, compared with 0.665 for DINOv2, and multi-view depth probing achieves 0.056 ARE in Table 3. These results suggest that the learned backbone contains useful information beyond what is needed for image reconstruction. The Casual YouTube ablation also provides evidence that less curated video can improve the system.
Critical weaknesses
W1. The specific benefit of nested motion-token compression is not fully isolated.
Table 5 shows that removing nested dropout substantially degrades camera estimation, supporting the importance of constraining motion conditioning. However, this ablation simultaneously changes the average token budget, prefix ordering, and exposure to absent motion/time conditioning during pretraining. It therefore leaves unclear which aspect produces the improvement.
A comparison with a small fixed motion-token budget and binary dynamic-state dropout under matched training conditions would clarify the methodological contribution. The existing results support the overall recipe, while these controls would strengthen the proposed compression-based explanation.
W2. The evaluation provides limited direct evidence for the accuracy of moving geometry.
Dynamic novel-view synthesis and the camera–time matrices provide encouraging evidence of controllable reconstruction. However, full-image rendering metrics can conceal errors around moving objects, particularly when static backgrounds occupy much of the image. Meanwhile, Tables 3 and 4 evaluate supervised probes on frozen features, rather than the original depths and motion masks produced by the reconstruction model.
Foreground-specific NVS metrics and a direct quantitative assessment of the native dynamic geometry would make the conclusions more convincing. The temporal scope should also be stated precisely: the current dynamic NVS protocol evaluates novel viewpoints at observed input times.
W3. Several reporting and implementation details require clarification.
The final cumulative configuration in Table 5 differs from the main results without an explanation of the experimental settings. For example, Wild-SLAM RRA@5 is 0.906 in Table 5 versus 1.000 in Table 1. The ablation labels also omit and , leaving their inclusion unclear. These differences may reflect separate training configurations, but the relationship between the experiments should be explicit.
Additionally, Eq. (16) directly compares single-view and multiview log depths despite the acknowledged monocular scale ambiguity. The authors should clarify whether normalization, alignment, or a shared learned scale convention makes these predictions compatible.
Questions for authors
-
Why does the final cumulative configuration in Table 5 differ from Tables 1–2, particularly Wild-SLAM RRA@5 of 0.906 versus 1.000? Which losses and training settings are shared?
-
How does nested dropout compare with a small fixed motion-token budget or binary dynamic-state dropout under matched conditions?
-
Can you provide foreground-specific NVS results or a direct quantitative assessment of the native moving geometry?
-
How are depth scales reconciled in Eq. (16), and how are time tokens obtained for timestamps without an observed image?
6 comments
W2 & Q3. Direct evidence for moving geometry
We agree that full-image metrics can hide errors on moving objects. We add foreground-specific NVS metrics and directly evaluate the native depth and motion masks produced by the model.
Foreground NVS. We evaluate on two additional datasets, DyCheck and DyNeRF (9 input frames, one camera per timestep), and report foreground metrics:
Each cell: DyCheck / DyNeRF.
| Method | Self-sup. | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR-fg ↑ | SSIM-fg ↑ | LPIPS-fg ↓ |
|---|---|---|---|---|---|---|---|
| NeoVerse | ✗ | 8.77 / 15.78 | 0.167 / 0.432 | 0.586 / 0.356 | 9.98 / 19.30 | 0.170 / 0.425 | 0.560 / 0.334 |
| C4G | ✗ | 12.93 / 18.95 | 0.298 / 0.469 | 0.631 / 0.356 | 11.00 / 19.17 | 0.217 / 0.458 | 0.664 / 0.394 |
| MoVieS | ✗ | 12.33 / 24.89 | 0.267 / 0.840 | 0.478 / 0.127 | 11.78 / 23.40 | 0.207 / 0.693 | 0.532 / 0.181 |
| Ours | ✓ | 13.54 / 20.54 | 0.265 / 0.523 | 0.546 / 0.251 | 11.92 / 21.09 | 0.204 / 0.567 | 0.580 / 0.243 |
We also report foreground metrics on NVIDIA Dynamic (Table 2 setting):
| Method | Self-sup. | PSNR-fg ↑ | SSIM-fg ↑ | LPIPS-fg ↓ |
|---|---|---|---|---|
| MoVieS | ✗ | 15.31 | 0.283 | 0.433 |
| C4G | ✗ | 14.10 | 0.202 | 0.589 |
| NeoVerse | ✗ | 15.29 | 0.279 | 0.363 |
| Ours | ✓ | 14.93 | 0.227 | 0.470 |
Ours is the only self-supervised method here, while all baselines rely on geometric supervision or pretrained geometric models. Even so, Ours achieves the best full-image and foreground PSNR on DyCheck, ranks second on DyNeRF, and outperforms C4G on all foreground metrics on NVIDIA Dynamic. These results show that our model is competitive with supervised methods on the moving regions, and we will add them to the revision.
Native predicted depth. We evaluate the raw depth output of the model (no probe) on TUM-Dynamic, over the full image and on the human foreground:
| Method | Self-sup. | AbsRel ↓ | δ1 ↑ | AbsRel-fg ↓ | δ1-fg ↑ |
|---|---|---|---|---|---|
| MonST3R | ✗ | 0.209 | 0.754 | 0.233 | 0.693 |
| NeoVerse | ✗ | 0.114 | 0.885 | 0.123 | 0.892 |
| E-RayZer | ✓ | 0.292 | 0.567 | 0.380 | 0.457 |
| Ours | ✓ | 0.206 | 0.673 | 0.265 | 0.569 |
Ours clearly outperforms the self-supervised E-RayZer and is on par with the supervised MonST3R in full-image AbsRel (0.206 vs. 0.209). NeoVerse and MonST3R achieve stronger foreground results, but both are supervised and built on large pretrained geometric foundation models, which greatly reduce reconstruction ambiguity. Without such supervision, 4D reconstruction from monocular video faces two coupled ambiguities: camera motion vs. scene motion, and the depth of a moving object vs. its motion. Our work jointly learns cameras and renderable dynamic geometry from uncalibrated monocular video, without geometric supervision or pretrained geometric models. Separating camera motion from scene dynamics is a central contribution, while our depth-consistency objectives, including L_sv, are designed to mitigate the depth–motion ambiguity in moving regions. The native-depth results show improvements over the self-supervised E-RayZer in both full-image and foreground metrics, while revealing a remaining gap to supervised methods in foreground geometry. Improving moving-object geometry is therefore an important next step within our self-supervised 4D reconstruction framework.
Native motion masks. We also evaluate the raw self-supervised masks produced by the model (no probe) on Kubric-4D, which provides per-object dynamic labels:
| Method | IoU ↑ | Precision ↑ | Recall ↑ | F1 ↑ | FPR ↓ |
|---|---|---|---|---|---|
| MonST3R | 0.007 | 0.008 | 0.056 | 0.014 | 0.265 |
| Ours w/o asymmetric cost | 0.193 | 0.217 | 0.606 | 0.303 | 0.169 |
| Ours | 0.287 | 0.388 | 0.528 | 0.418 | 0.069 |
Since these masks come directly from the reconstruction model, they complement the probing results in Table 4. Self-supervised feed-forward baselines such as E-RayZer do not produce motion masks, so we use MonST3R as the reference. Its masks often include static structure such as the ground, which explains its very low precision. The asymmetric cost targets exactly this failure mode. By favoring static explanations, it lowers the false-positive rate from 0.169 to 0.069 and raises precision from 0.217 to 0.388, with a moderate drop in recall. We will include qualitative mask comparisons in the revision.
Temporal scope. This is a fair point. Our model reconstructs one scene state per input frame, in the same per-timestep manner as MonST3R. The dynamic NVS protocol therefore evaluates novel viewpoints only at the observed timestamps, and we will state this scope explicitly in the paper.
W3 & Q1. Table 5 vs. Tables 1–2, and the included losses
We apologize for the missing description. The full-model row of Table 5 is the same model as in Tables 1–2, and the discrepancy comes from the evaluation protocol. Tables 1–2 must use settings that all baselines support. Table 5 compares only our own variants and therefore uses the protocol matching our training setting:
- Camera. Table 5 runs non-overlapping 9-frame windows and computes RRA/RTA over all pairs within each window. Table 1 uses windows that share boundary frames and reports adjacent-pair metrics, which all baselines support. The different windowing also changes the intrinsics estimate and hence AREf.
- NVS. Table 5 uses 9 input frames, our training length, so that the results are not affected by running on inputs longer than those seen in training. Table 2 uses 12 frames to match the baselines.
All rows of Table 5 share the same protocol, so the relative comparisons hold. In the revision, we will describe this protocol of Table 5 and explain how its numbers relate to those in Tables 1–2.
Regarding the losses: L_distill, L_reg and the context-only Gaussian pass came from early experiments and are always enabled in Stage 2. In Table 5, they are included in the L_exp row and in every row below it. We apologize for omitting them from the labels and will fix this.
Q4 (a). Depth scale in Eq. (16). We apply no explicit normalization or alignment. Instead, the model is encouraged to adopt a shared learned scale convention for a given scene across different inference passes. Two mechanisms enforce this:
- Eq. (16) directly penalizes any scale difference between single-view and multi-view log-depths, so the model is pushed to predict the same scale in both settings.
- As described in Appendix C.1, the context-only pass predicts Gaussians from the initial (not bootstrapped) camera tokens to prevent camera leakage. These Gaussians are then unprojected with the cameras predicted by the bootstrap pass. This requires the depth scale of the context-only pass to agree with the translation scale of the bootstrap cameras.
Q4 (b). Time tokens for unobserved timestamps. As noted under W2, our current formulation reconstructs the scene state at observed timestamps. Each time token is predicted from the image observed at that timestamp, so we do not obtain time tokens for unobserved times, and temporal interpolation is not constrained during training. Since the time token has a single degree of freedom, interpolating between neighboring time tokens is a natural extension and an interesting direction for future work.
In addition, we would like to share a related result. Although temporal interpolation is not trained, the backbone already learns dense cross-time correspondence, which we believe is a useful basis for such an extension. A frozen-backbone optical-flow probe with probe heads of comparable size shows this:
Each cell: Sintel / Spring.
| Backbone | EPE ↓ | Fl-all ↓ |
|---|---|---|
| CroCo v2 | 23.62 / 4.05 | 0.815 / 0.285 |
| E-RayZer | 24.24 / 5.72 | 0.814 / 0.449 |
| VideoMAEv2 | 17.72 / 2.73 | 0.543 / 0.179 |
| Ours | 14.48 / 2.58 | 0.570 / 0.178 |
Ours achieves the lowest EPE on both Sintel and Spring and the best Fl-all on Spring, and is behind VideoMAEv2 only in Sintel Fl-all. It also clearly outperforms CroCo v2 and E-RayZer. This suggests that our features capture not only viewpoint changes but also how scene content moves over time.
We thank you again for your valuable feedback and hope our responses address your concerns. We are incorporating these results into the revised manuscript and would be happy to discuss any further questions.
Dear Reviewer nSRp,
Thank you for your careful review and for recognizing our design choices, in particular the context-only Gaussian pass and the shared explicit camera conditioning. We are glad that the failure-mode visualizations helped convey our motivation. Your suggestions are very helpful, and we are revising the paper accordingly. We apologize for the late reply, as we wanted to run the additional experiments needed to answer your questions as thoroughly as possible. Our responses are below.
W1 & Q2. Isolating the benefit of nested motion-token compression
Thank you for suggesting these two very constructive controls. Binary dynamic-state dropout is similar to the strategy used in RayDer. We trained lightweight models for the three variants under identical settings, using 4 motion tokens for the small fixed budget, and probed the learned features. Together, these controls separate the three factors you identified: the token budget, the exposure to absent motion conditioning, and the ordered (prefix) compression.
Camera probe:
Flow probe on Spring:
Token budget and absent motion conditioning (cameras). The small fixed budget uses only 4 motion tokens, yet it gives clearly the worst camera features. A small budget alone therefore does not remove the shortcut. Because the motion tokens are always available, they always provide a path that bypasses the camera and encodes the target image directly, however small the budget. Binary and nested dropout both close this path by regularly dropping the motion pathway, which forces the camera pathway to explain viewpoint change. As a result, both learn clearly better camera features.
Ordered compression (motion). Binary and nested dropout both expose the model to absent motion conditioning, but they differ when the motion tokens are present. Binary dropout leaves them uncompressed and unordered. Nested dropout instead compresses information toward the leading tokens, which further limits target-appearance leakage and promotes correspondence learning. This is reflected in the flow probe, where nested dropout is clearly best (EPE 2.29 vs. 2.69). In the style-bootstrap experiment of Fig. 2, the binary-dropout variant also fails to predict context-consistent colors, and we will add this visualization to the revision.
Finally, different dynamic scenes need different numbers of motion degrees of freedom. Nested dropout does not commit to a fixed budget in advance. It only requires the motion information to be as compressed as possible, which a fixed budget cannot provide.