acceptodds
Under review as a conference paper at ICLR 2027

Old-School Warping Shines for Modern Self-Supervised 3D Reconstruction

Abstract

Humans can develop a rich understanding of the 3D world with little to no explicit geometric supervision. Inspired by this ability, we show that a simple multi-view Transformer can learn strong 3D reconstruction capabilities through self-supervision with an "old-school" image warping loss. Our model, Warp3R, jointly predicts depth maps and camera poses and is trained using photometric and geometric consistency across views. It substantially outperforms state-of-the-art self-supervised methods on camera pose estimation across diverse datasets and improves self-supervised depth estimation. Extending supervised training with our objective on unlabeled data further improves both pose and depth estimation. Our results highlight the effectiveness of classical geometric principles for learning 3D reconstruction with modern networks. Code will be released to facilitate future research

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised, 3D, multi-view-geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s5UJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a warping-based training objective for self-supervised 3D reconstruction, where both the camera poses and depth maps of the given input views are predicted without intermediate 3D representations like 3DGS needed. Experiments show that it outperforms other state-of-the-art self-supervised reconstruction paradigms in camera pose estimation and depth prediction.

Critical strengths
  • The idea itself is technically sound, simple, and clean.
  • The empirical results, particularly the zero-shot camera estimation, are strong.
  • The ablation study is broad and informative, in which many designs are investigated.
Critical weaknesses
  • Despite the simplicity, the strong geometric inductive bias underlying reprojection also limits the upper ceiling of Warp3R. Specifically, the reprojection assumes that the input views are from a single rigid scene. Other complex effects (e.g., object deformation, illumination change, view-dependent appearance) might not be properly modelled. In other words, while the method is very simple and effective at modelling rigid geometric correspondence, it sacrifices its capability to handle complex and unavoidable effects in real scenes beyond geometric correspondence and thus results in a clear boundary in applicability.
  • E-RayZer and NAS3R learn 3DGS to support novel-view synthesis rather than directly optimizing geometric reconstruction like camera poses and depths. In other words, the ultimate goal of E-RayZer and NAS3R is to improve rendering quality at novel views, which does not necessarily mean explicitly targeting more accurate camera poses and depths like the structure-from-motion methods. Instead, Warp3R uses a reprojection-based objective, directly and explicitly targeting camera pose/depth estimation accuracy. While the advantages are well acknowledged, they do not necessarily imply that warping is a superior reconstruction paradigm overall. It might show that "in the accurate geometric reconstruction case where the reprojection-based objective is favored, the method performs better than other methods that are not primarily designed for the task but prioritize perceptual quality". I do not consider this a critical weakness of the method itself, but rather a concern about the experimental setting.
Questions for authors
  • NAS3R enables self-supervised learning from scratch. Can Warp3R realize this without any prior? The failure may indicate the optimization difficulty or an intrinsic limitation of the reprojection-based supervision and provide insights for future improvements.
  • Can the authors also provide the 3D-point evaluation results like VGGT? This can greatly strengthen the claim of "3D reconstruction".
  • Can the authors provide any discussion or analysis regarding the boundary or failure patterns when the rigid-scene reprojection is violated?|
  • Currently, Warp3R explicitly uses temporal adjacency by setting the temporal neighbourhood to adjacent views. Can this be extended to unordered views or randomly sampled frames? This directly answers whether Warp3R is a multi-view reconstruction method or specifically a video reconstruction method.
  • Given the same data budget, how is this self-supervised learning objective compared with the supervised counterpart like VGGT?
3 comments

Thank you for highlighting the "strong" zero-shot camera results. E-RayZer and NAS3R address both geometric reconstruction and novel-view synthesis, which we agree are distinct evaluations. Tables 1–2 compare camera accuracy using their established evaluation protocols and matched reconstruction-training datasets: DL3DV for E-RayZer and RE10K for NAS3R. These results demonstrate the effectiveness of a simple warping-based system for geometric reconstruction.

1. “NAS3R enables self-supervised learning from scratch. Can Warp3R realize this without any prior?”

Warping alone has not reliably converged from random initialization in our runs. Strong results with alternative self-supervised encoders (Table 3k–m) show that successful optimization is not specific to DINOv2 initialization.

2. “Can the authors also provide the 3D-point evaluation results like VGGT?”

Thank you for suggesting this experiment. We have not performed a quantitative point-cloud evaluation of the submitted model. Tables 1–2 and Figure 5 assess pose and depth; Figures 3 and 7 show qualitative point-cloud reconstructions.

3. Discussion of “object deformation, illumination change, view-dependent appearance”

Like E-RayZer (their §5), Warp3R assumes a static scene. Object motion can create image changes a rigid-scene model cannot explain; strong illumination changes and view-dependent appearance can cause photometric errors even with correct geometry. We will discuss these limitations and directions for future work in the final version.

Our results show that training on real-world DL3DV videos with natural lighting and appearance variation yields strong zero-shot pose results (Table 1); Figure 10 illustrates accurate poses in reflective scenes present in our evaluations.

4. “Warp3R explicitly uses temporal adjacency … Can this be extended to unordered views or randomly sampled frames?”

Our training recipe uses temporal continuity in static-scene video (our paper, lines 253–258). Training without frame-index embeddings gives a test-time permutation-equivariant variant with 84.4 DL3DV RPA@5 (Table 3n). Expanding the temporal warping neighborhood from ±1 to ±2 or ±3 retains comparable or improved DL3DV pose accuracy (Table 3f–g). We have not tested random-frame training with the default configuration.

5. Comparison with VGGT-style supervision: “Given the same data budget”

Figure 6’s fully supervised and fully self-supervised endpoints use the same number of frames from the same datasets, with architecture, initialization, batch size and updates fixed. Self-supervision yields higher mean RPA@5; the VGGT-style supervised counterpart yields slightly lower depth error.

We thank the authors for the brief response. It addresses some of my concerns while leaving W1 and W2 open. The dependency on DINOv2 initialization also reflects limitations. I retain my current score.

Thank you for the follow-up and for acknowledging the concerns addressed by our response. Regarding the remaining concerns, we would like to elaborate on W1 and W2 and clarify the role of initialization.

W1. “illumination change, view-dependent appearance”

We agree that strong illumination changes and view-dependent appearance can undermine photometric agreement even when the geometry is correct, and can be a limitation.

This actually gets to the heart of why we find the result interesting: these classical losses nevertheless train modern VGGT-like systems effectively on real-world DL3DV video containing these appearance changes, yielding strong zero-shot pose accuracy (Table 1), with qualitative examples of accurate poses in reflective evaluation scenes (Figure 10). Figure 6 also shows that purely self-supervised training achieves higher mean pose accuracy (RPA@5) on those same real-world benchmarks than the supervised counterpart, with datasets, frame budgets, and architecture matched.

We believe this is useful for the community: it establishes an effective training recipe that others can build on, and a concrete basis for investigating how far classical warping objectives can be taken in modern reconstruction systems.

W1 (continued). “object deformation” and dynamic objects

Independent object motion and deformation are not modeled by our static-scene formulation, and we agree that this is a current limitation. We will clarify this in the paper.

We view the demonstrated effectiveness of this training recipe as a useful milestone that can inform future efforts to address dynamic scenes.

W2. “E-RayZer and NAS3R learn 3DGS to support novel-view synthesis”

Thank you for highlighting the distinction between novel-view image quality and geometric accuracy. We agree that the additional rendering goal can motivate different design choices.

Both E-RayZer and NAS3R evaluate camera estimation and, like Warp3R, learn cameras and explicit geometry through image reconstruction without camera or depth annotations. E-RayZer says it “operates directly in 3D space, performing self-supervised 3D reconstruction with Explicit geometry.”

Tables 1–2 compare this shared task using matched reconstruction training datasets and their pose evaluation protocols, with all methods evaluated on the same frames.

I do not consider this a critical weakness of the method itself, but rather a concern about the experimental setting.

We appreciate your distinguishing the method from the interpretation of these comparisons. Under these shared conditions, the complete Warp3R system achieves the strong camera accuracy.

Initialization. “dependency on DINOv2 initialization”

Table 3(k–m) shows strong results with MoCo v3, MAE, and Pixio, demonstrating that successful optimization is not specific to DINOv2. In each case, we repurpose blocks pretrained on individual images for attention across frames from initialization (lines 152–156), extending their use beyond the single-image attention setting of pretraining.

Though both the initialization and the subsequent reconstruction training are fully self-supervised, we agree that requiring pretrained self-supervised initialization remains a limitation.

In light of these results, we would appreciate your reconsidering the weight placed on this initialization limitation when assessing the overall contribution of this work.

Reviewer_L6YdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Warp3R proposes a self-supervised approach for training feed-forward 3D reconstruction networks. The method revisits ideas from pre-ViT approaches, such as Monodepth, Monodepth2, and SC-SfMLearner, and adapts them to train a modern multi-view Transformer architecture.

The core idea is simple: a feed-forward reconstruction network predicts camera poses and depth maps. Assuming that the scene is static and that the input images have sufficient overlap (as is typically the case for neighboring video frames), one image can be warped to partially reconstruct another. This enables the use of a photometric consistency loss to train the network in a self-supervised manner. Warp3R additionally introduces a geometric consistency loss based on warping the predicted depth maps.

In the experiments, Warp3R is fairly compared against state-of-the-art methods E-RayZer and NAS3R, significantly outperforming both in camera pose estimation. Warp3R also frequently improves upon E-RayZer in depth estimation.

The paper includes an extensive ablation study to evaluate the contributions of the different components.

Finally, the authors investigate a semi-supervised setting, combining the proposed self-supervised objectives with VGGT's supervised losses to leverage additional unlabeled data. They demonstrate that this approach improves both camera pose and depth estimation.

Critical strengths
  1. Simplicity and effectiveness -- Warp3R is a remarkably simple method, particularly compared to recent state-of-the-art approaches such as E-RayZer and NAS3R. Despite its simplicity, it significantly outperforms these methods, especially in camera pose estimation. This is a compelling result.

  2. Convincing experimental evaluation -- The experimental comparisons appear fair and the results are convincing. The extensive ablation study further strengthens the empirical evaluation and helps establish the contributions of the different components.

Critical weaknesses
  1. Limited discussion of the underlying motivations and optimization challenges -- The paper is largely written as a description of the proposed method, which itself revisits ideas from Monodepth, Monodepth2, and SC-SfMLearner. While the methodology is clearly presented, the underlying motivations and implications of several design choices are insufficiently discussed. The presentation often feels somewhat procedural, with components introduced mainly because they were used in earlier works, rather than through a discussion of why they are appropriate in the present setting. For instance, the use of minimum aggregation for the geometric consistency loss in Eq. (7) is not really motivated or discussed. More fundamentally, the proposed approach relies on learning through image warping, where the gradients of the photometric consistency loss with respect to the predicted geometry are propagated through bilinear interpolation. One might expect this to make optimization challenging, particularly at the beginning of training when the predicted camera poses and depth maps are inaccurate. However, the paper does not discuss this issue or provide insights into the early stages of optimization.

  2. Insufficient discussion of the assumptions underlying the method -- The proposed approach fundamentally relies on sufficient overlap between the input images and limited appearance changes across views, so that meaningful photometric and geometric consistency signals can be obtained through warping. These assumptions are important for understanding the applicability and robustness of the method, yet they receive only a very brief discussion (lines 253–257). A more thorough analysis of how the method behaves when these assumptions are not satisfied would strengthen the paper.

  3. Lack of discussion of limitations -- I could not find a dedicated discussion of the limitations of the proposed approach. In particular, the reliance on static-scene assumptions is an important limitation of warping-based self-supervision. I believe this limitation, along with the assumptions discussed above, deserves explicit acknowledgment and discussion.

Questions for authors
  1. Related work. In the section on self-supervised visual representation learning, I noticed that several recent works are missing, including MuM, Muskie, and Gekko. Is there a particular reason why these methods are not considered relevant related work?

  2. Depth warping. What interpolation scheme is used to warp the depth maps in Eq. (5)?

  3. Training without pretrained initialization. Have the authors experimented with training Warp3R from scratch, i.e. without DINOv2 initialization? If so, does training converge more slowly, or does it fail altogether? Understanding the role of pretrained initialization would be particularly interesting in the context of the proposed self-supervised objective.

  4. Early training dynamics. I would be interested in understanding what happens during the initial stages of training, when the predicted camera poses and depths are still inaccurate. In particular, since the photometric loss propagates gradients through bilinear interpolation, how does the network obtain a useful learning signal when the initial warps are highly inaccurate? An analysis or visualization of the early training dynamics would be valuable.

  5. Depth estimation comparisons. Is there a particular reason why NAS3R is not included in the depth estimation experiments?

  6. Limitations. Could the authors include a more detailed discussion of the limitations of the proposed approach, particularly regarding dynamic scenes, limited view overlap, and appearance changes across images?

4 comments

Thank you for the constructive questions.

Q1. Related work: “MuM, Muskie, and Gekko”

We will add these works to the related-work discussion.

Q2. “What interpolation scheme is used to warp the depth maps in Eq. (5)?”

We bilinearly interpolate predicted z-depth.

Q3. “Have the authors experimented with training Warp3R from scratch, i.e. without DINOv2 initialization?”

Warping alone has not reliably converged from random initialization in our runs. Strong results with alternative self-supervised encoders (Table 3k–m) show that successful optimization is not specific to DINOv2 initialization.

Q4. Early training: “how does the network obtain a useful learning signal …?”

In the paper’s base configuration, total loss and pose error rate on training clips (the complement of RPA@5) drop quickly and smoothly from the outset without prolonged plateaus or abrupt jumps. This pattern is consistent across runs; we will include these training curves in the release. Our 3×3 SSIM term couples neighboring pixels, providing spatial context alongside bilinear warping.

Geometric consistency: “minimum aggregation … in Eq. (7)”

The minimum selects the valid source with the smallest geometric residual for each pixel, without requiring consistency with every source when visibility differs. We will explain this rationale explicitly.

Q5. “Is there a particular reason why NAS3R is not included in the depth estimation experiments?”

Our main model is trained on DL3DV, so depth evaluation focused on a matched-DL3DV-training comparison, for which we used E-RayZer’s released checkpoint.

Q6. Discussion of “dynamic scenes, limited view overlap, and appearance changes across images”

Warp3R assumes a static scene, as does E-RayZer (their Section 5). Object motion can create image changes a rigid-scene model cannot explain; limited view overlap reduces the available warping supervision; strong illumination changes and view-dependent appearance can cause photometric errors despite correct geometry. We will add a dedicated discussion of these limitations.

Training on real-world DL3DV videos with natural lighting and appearance variation yields strong zero-shot pose results (Table 1). Figure 5 evaluates depth across camera baselines, and Figure 10 gives reflective-scene examples.

Thank you for your responses. However, I find them rather brief, and I would appreciate a more detailed discussion of the points raised in my review. In particular, I encourage the authors to elaborate further on their answers to the specific questions, as well as to address the identified weaknesses more thoroughly. Providing additional explanations/clarifications would help me better assess the extent to which my concerns have been addressed.

Thank you for following up. We expand below on the design rationale, early optimization, and assumptions and limitations highlighted in your review.

Q4. Early training: “how does the network obtain a useful learning signal …?”

Our intuition is that warping creates a geometric bottleneck. The model reconstructs a reference image by resampling observed source views through predicted camera parameters and depth. A small set of camera parameters governs all pixels, while each pixel has one predicted depth value; geometric consistency further couples these depths across views. As with an autoencoder bottleneck, the reconstruction objective encourages this constrained representation to capture useful structure.

Because the same camera parameters affect many reprojections, comparisons favoring a consistent geometric change can reinforce one another in a camera update. We expect this aggregation, together with SSIM’s spatial context and filtering of unreliable comparisons, to help provide useful learning signals even when individual photometric gradients are noisy.

To elaborate on the “underlying motivations and implications of several design choices,” we view these components as ways to reduce noise in the learning signal:

  • SSIM: We first form the warped image by bilinear sampling from 2×2 source neighborhoods, then apply SSIM to overlapping 3×3 patches of the warped and reference images. These windows make the gradient at each warped pixel depend on neighboring intensities through shared patch statistics; the gradients then propagate through the bilinear sampler to the predicted geometry. Our intuition is that this shared spatial context makes updates less sensitive to isolated intensity fluctuations and noisy individual pixel comparisons. Q6 discusses the complementary role of SSIM’s normalized structure term under appearance changes.
  • Auto-masking: Following Monodepth2, we retain pixels only when warping improves the photometric match over the unwarped comparison. We view this as a way to limit uninformative gradient contributions from comparisons where the predicted warp offers no photometric improvement, including stationary-camera cases.
  • Geometric consistency and “minimum aggregation … in Eq. (7)”: Following Bian et al. (2019), we encourage predicted depths to agree across views after reprojection. For source aggregation, we select the smallest normalized depth discrepancy to reduce conflicting supervision from inconsistent source comparisons. Interpolation near depth boundaries can mix depths from different surfaces, and occlusions can make source views disagree. This selection limits the influence of such outlier residuals while retaining a cross-view consistency constraint.

As noted in our previous response, the paper’s base configuration shows a rapid, smooth decrease in total loss and pose error rate on training clips (the complement of RPA@5) from the outset, without prolonged plateaus or abrupt jumps. This pattern is consistent across runs; we will include these training curves in the release.

Table 3 shows that individual components can improve accuracy while strong performance persists across the tested pretrained initializations and ablations. For example, auto-masking improves pose accuracy, but removing it still leaves a strong model.

We will expand the paper’s discussion of the design rationale and intuitions.

Q6. Discussion of “dynamic scenes, limited view overlap, and appearance changes”

We agree that these assumptions and their practical consequences deserve explicit discussion:

  • Dynamic scenes: Our training and evaluation target static-scene reconstruction. Independent object motion introduces image changes that this rigid-scene model cannot explain, so reprojection errors in moving regions cannot be attributed to camera or depth errors alone.
  • Limited overlap: Less shared visible content leaves fewer pixels available for reliable warping supervision. This motivates neighboring training views; though Table 3 also shows strong performance with expanded temporal neighborhoods. Figure 5 evaluates depth across camera baselines: gains are clearest at small to moderate baselines on 7-Scenes and Neural RGB-D, and vary at wider baselines.
  • Appearance changes: SSIM’s structure term compares local patterns after normalizing brightness and contrast, allowing shared structure to contribute when raw intensities differ. Strong illumination changes and view-dependent reflections can still produce photometric errors even when the geometry is correct.

The submitted results show useful learning under real-world lighting and appearance variation: training on DL3DV yields strong zero-shot pose results (Table 1), and Figure 10 illustrates accurate poses in reflective scenes.

We will add a dedicated discussion of these limitations to the paper.

Dear authors,

Thank you for the additional explanations. I have to admit that it is still unclear to me how the network learns through backpropagation in this setting. I find this especially puzzling given that the network learns when initialized from DINO and fails to learn when trained from scratch. However, I see this as an interesting open question and will keep my initial rating.

Best regards,

Reviewer_z8RpOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper revisits image warping for self-supervised multi-view 3D reconstruction. The proposed Warp3R uses a pretrained vision Transformer with alternating local and global attention to predict camera parameters and depth maps. It is trained with photometric warping and scale-normalized geometric consistency, with validity, occlusion, and auto-masking rules. The paper shows substantial camera-pose improvements over E-RayZer and NAS3R, and provides depth evaluation and ablation studies on the losses, architecture, initialization, and frame embeddings. It also studies replacing part of a fixed supervised training budget with unlabeled data using the warping objective.

Critical strengths
  1. Strong results with a simple objective. I appreciate the finding that classical warping can still work well with a modern multi-view backbone. In Table 1, the DL3DV RPA@5 improves from 72.1 of the reproduced E-RayZer to 92.5, and several unseen datasets also have large improvements. The comparison trained on RE10K in Table 2 is useful to address the training-data difference with NAS3R. The simple objective itself is appealing to me.

  2. Detailed ablation studies. Table 3 studies the losses, masking, attention frequency, initialization, and frame embeddings, and reports the variation over five runs for the default baseline. The semi-supervised experiment keeps the architecture, batch size, and number of updates the same. The appendix also explains the released and reconstructed evaluation protocols. These details are useful for understanding and checking the results.

Critical weaknesses
  1. The reason for the improvement is not fully clear. The results show that the full Warp3R system works well. However, E-RayZer and NAS3R use different architectures and training formulations, while Table 3 mainly studies changes inside the warping-based system. I did not find a comparison that keeps the backbone, initialization, data, and optimization budget the same, and only changes between warping and view-synthesis supervision. Since the paper focuses on the effectiveness of old-school warping, I would prefer to see this comparison to understand the contribution of the objective. The simplicity of warping is not a weakness to me.

  2. The large-baseline results need more explanation. In Table 1, Warp3R is worse than the reproduced E-RayZer on BlendedMVS at RPA@15 and RPA@30, although it improves most other results. The depth curves also show that the improvement is not consistent at wider baselines. I noticed that Appendix Table 5 only uses seven BlendedMVS scenes, so I would not draw a strong conclusion from this alone. However, it would be useful to understand under which overlap or baseline conditions the direct warping becomes less effective.

Questions for authors
  1. Is there a comparison between warping and view-synthesis supervision with the same backbone, training data, and computation budget? If not, please clarify that the current results show the gain of the full system, while the contribution of the loss alone is not isolated.

  2. Would it be possible to analyze the pose and depth errors according to view overlap or camera baseline, including the BlendedMVS cases? Table 1 gives 44.4/53.3 for Warp3R and 47.6/69.2 for reproduced E-RayZer at RPA@15/30. An explanation of these cases would help understand the applicable settings of the method.

  3. How does the fraction of valid and auto-masked pixels change during training, especially for wide-baseline clips? Since pixels without an eligible source are excluded, I am interested to know whether degenerate camera/depth predictions or almost empty supervision happen, and how often training fails across different runs.

2 comments

Thank you for the careful assessment.

1. Comparing warping and view-synthesis supervision with “the backbone, initialization, data, and optimization budget the same”

Our central finding is that classical warping provides an effective training objective for a modern multi-view reconstruction system. We match training datasets and evaluation frames, comparing complete systems in Tables 1–2. We have not performed the requested loss-only comparison, so the cross-method gains do not isolate the contribution of the objective. We will clarify this scope.

2. Pose and depth errors by “view overlap or camera baseline, including the BlendedMVS cases”

Figure 5 evaluates depth across camera baselines: gains are clearest at small to moderate baselines on 7-Scenes and Neural RGB-D and vary at wider baselines. Larger viewpoint changes can reduce overlap and increase occlusion, leaving fewer usable comparisons for warping. This motivates neighboring training views. The aggregate BlendedMVS pose scores do not establish whether overlap explains the gap to E-RayZer.

3. “How does the fraction of valid and auto-masked pixels change during training, especially for wide-baseline clips?”

In the paper’s base configuration, all mask statistics show stable, smooth, monotonic trends throughout training that are quick to converge to their final values. The fraction of pixels with a usable source increases, auto-masking rejects a decreasing share of valid pixels, and the fraction retained for the loss reaches about 80%. We observed no failures to converge across runs of this base configuration. These logs are not stratified by camera baseline.

Thank you for the helpful clarifications. I appreciate the claims. I will maintain my rating of clear acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.