acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Asymmetry in Self-Supervised 3D Geometry Learning

Abstract

Self-supervised learning relies on asymmetry between what a model observes and what it must explain. Recent methods learn 3D geometry from unposed images by predicting cameras and scene structure, then rendering held-out views. To avoid trivial solutions, they separate camera and scene branches or restrict interaction between context and held-out views, sacrificing cross-view correspondence. We revisit this trade-off with geometric accuracy as the objective. When a held-out view lies between context views, the context already constrains the geometry it observes. We show that excluding such views only from the 3D representation prevents trivial solutions while allowing a shared backbone to use all views for correspondence, improving depth and pose estimation over restricted designs. We further exploit view density: sparse and dense observations depict the same scene but provide different amounts of correspondence. Aligning sparse-context features with dense-context features supplies a geometric training signal alongside rendering, without a pretrained geometric teacher. We additionally align features with a frozen vision foundation model, which brings further gains. None of these signals require geometric annotation, and the resulting model improves geometry estimation over prior self-supervised methods. We analyze each component in controlled experiments validating the effectiveness of our proposed methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-supervised Learning, 3D reconstruction, Camera pose estimation, Depth estimation, Novel-view synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GCuiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper follows recent approaches that learn 3D geometry from unposed images by jointly predicting camera parameters and scene structure, followed by rendering held-out views for self-supervision. Unlike these methods, the proposed approach introduces self-supervised asymmetry during 3D scene construction rather than feature extraction. It further exploits the asymmetry between sparse and dense representations, and incorporates feature alignment with a frozen vision foundation model, leading to additional performance gains.

Critical strengths
  • The paper is clearly written and easy to follow. The figures are well designed and effectively illustrate the key ideas and experimental results.
  • The main strength of the paper is the idea of introducing asymmetry in the 3D representation space rather than during feature extraction. This is both novel and well motivated. When the held-out views are interpolated between the context views, they primarily provide additional correspondences to improve geometric accuracy instead of introducing new scene content. Consequently, it is only necessary to prevent a held-out view from contributing to its own 3D representation during rendering. Based on this insight, the authors employ full-view attention during feature computation while excluding the held-out views from the 3D scene construction, enabling richer feature interactions without introducing information leakage.
Critical weaknesses
  • The proposed full-view strategy relies on the assumption that held-out views do not introduce new scene content. While this is reasonable for simple camera trajectories, it may not hold for datasets with complex viewpoint changes, where even interpolated views can reveal previously unseen regions. It would be helpful for the authors to discuss whether the proposed strategy remains valid under such settings and what limitations it may have.
  • The comparison with prior methods raises some fairness concerns. The proposed model is initialized with the Pixio (Base) visual encoder and further benefits from semantic guidance provided by both DINOv3 and Pixio, all of which are strong 2D pretrained models. In contrast, the compared baselines, including NAS3R and Erayzer, are trained from random initialization. Although Erayzer leverages pretrained priors for view sampling, these priors are not directly incorporated into the reconstruction model itself. As a result, it is difficult to disentangle the gains brought by the proposed method from those attributable to the powerful pretrained priors. This concern is further supported by Table 7, which shows that the pretrained components contribute substantially to the overall performance. It would therefore strengthen the paper if the authors could provide comparisons under a setting without pretrained initialization or semantic guidance.
  • Although the paper primarily focuses on geometric evaluation, it would be valuable to include a more comprehensive evaluation of NVS performance. Self-supervised 3D Gaussian representations are also valuable for rendering applications. In addition to comparisons with self-supervised 3DGS methods, it would be helpful to include comparisons against conventional GS methods, including both pose-required and pose-free approaches, to better position the proposed method with respect to the broader literature.
Questions for authors
  • The proposed full-view strategy appears to be particularly well suited to pixel-aligned Gaussian representations. It would be interesting to discuss whether this idea could be extended to non-pixel-aligned paradigms, such as C3G [1] or TokenGS [2], even though these methods are not designed for self-supervised learning. A discussion of the generality of the proposed strategy would help clarify its broader applicability.
  • According to the supplement, the training views are sampled solely based on frame intervals, I am curious about how stable is the optimization for difficult cases?

[1] C3G: Learning Compact 3D Representations with 2K Gaussians, CVPR 2026 [2] TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens, CVPR2026

8 comments

We thank the reviewer for the careful reading and constructive questions. We address each point below; all new results will be added to the revision.

(W1 & Q2) Validity of the full-view assumption under complex viewpoint changes, and optimization stability

What the assumption requires. A target view may attend to the context views, but it is never rendered from its own prediction: Gaussians come from the context views only. Target content not covered by the context therefore cannot be reconstructed, and its photometric error cannot be reduced. Our assumption is thus not that targets contain no unseen content, but that such content is a small fraction of each target. As it grows, these pixels act as noise in the rendering loss.

Stress test. We widen the frame interval between sampled views during training and at test time; wider intervals mean larger viewpoint changes and more unseen content in each target. We also test a simple remedy, a per-pixel confidence-weighted photometric loss c⋅∥I^−I∥2−αlog⁡cc \cdot \lVert \hat{I} - I \rVert^2 - \alpha \log c with α=0.02\alpha = 0.02. A separate DPT head with the same architecture as the Gaussian head predicts a confidence c∈(0,1)c \in (0, 1) (sigmoid) for each target pixel, so the model can down-weight pixels the context cannot explain. Results on DL3DV, averaged over 5/7/9/11 views (evaluation span in frames):

Training intervalConf. lossR3 (48)A30 (48)R3 (64)A30 (64)
32–48 (default)66.688.052.679.5
32–48 (default)✓72.089.557.881.6
48–64 (wide)40.170.428.758.9
48–64 (wide)✓67.788.155.080.0

R3: RPA@3; A30: AUC@30.

  • Wider spans at test time. The default model degrades moderately on the wider span without failing.
  • Wider intervals during training. With the plain photometric loss, training becomes unstable, and the model is worse than the default even on the wider span it was trained for. This is the limitation the reviewer points out: when targets contain too much unseen content, the rendering loss alone is no longer a reliable signal.
  • Confidence weighting. It largely recovers this failure and also improves the default setting, suggesting that unexplainable content is present even with moderate intervals.

The predicted confidence is low on content absent from the context views, such as objects entering the frame, and on thin, high-frequency structures that are hard to reconstruct. We will add this analysis, visualizations of the confidence maps, and a discussion of this limitation to the revision.

(W2) Fairness of comparison with respect to pretrained priors

We first clarify that our main model uses a single pretrained model, Pixio, both for initialization and as the semantic guidance teacher; DINOv3 appears only as an alternative in the ablation (Tab. 7).

We agree that the pretrained prior contributes substantially (Tab. 7) and that the comparisons with NAS3R and E-RayZer in Tab. 1–2 are not prior-matched. Our controlled comparison (Sec. 3.2, Fig. 3) isolates the design choice from the prior. All variants use the same pretrained encoder, backbone, data, and training schedule, and none uses semantic guidance or context alignment. The separated variant follows E-RayZer and even has more capacity (an additional 12-layer transformer). Under these matched priors, full-view outperforms both alternatives in Fig. 3, and the ranking holds after supervised fine-tuning of each variant (50K iterations, following Tab. 4):

Init. (→ SFT 50K)D-R3D-R5D-R15D-A30S-R3S-R5S-R15S-A30N-R3N-R5N-R15N-A30
Separated (E-RayZer design)48.273.196.386.71.34.630.530.016.536.880.270.6
Masked-view65.284.898.190.71.35.937.335.420.044.788.477.2
Full-view (ours)73.489.398.692.43.410.144.741.433.362.093.681.9

D/S/N: DL3DV/ScanNetv2/NRGBD; R3/R5/R15: RPA@3/@5/@15; A30: AUC@30.

The advantage of our design therefore does not come from the prior.

We acknowledge that this comparison is prior-matched rather than prior-free. Tab. 7 reports our model without pretrained initialization, but this analysis-scale model is not directly comparable to the baselines in Tab. 1–2, which use full-scale training. In the revision, we will state this explicitly for Tab. 1–2 and refer to Fig. 3 and Tab. 7, which separate the contribution of our design from that of the prior.

(W3) NVS evaluation and comparison with conventional GS methods

Our work treats NVS as a proxy task for self-supervised geometry learning, not as the end goal. Rendering provides a training signal without labels, and we evaluate the geometry it teaches, namely camera pose and depth. This is also what makes our full-view design possible. Since rendering quality is not the target, we let target images inform the context features to learn richer cross-view correspondences, and only prevent each target from being rendered from its own prediction. As a result, the rendering metrics of the full-view model do not measure novel view synthesis, so we gray them out rather than compare them with GS methods.

To still assess rendering without leakage, Appendix A.3 trains with full-view attention for the first 80% of iterations and switches to masked attention for the rest (Full-view → Masked-view). We compare it with the leakage-free designs trained for the same total schedule (DL3DV, averaged over 5/7/9/11 views):

MethodPSNR↑SSIM↑LPIPS↓
Separated19.880.5630.367
Masked-view20.350.5990.342
Full-view → Masked-view20.640.6200.322

The switched model renders best on all metrics and at every view count, suggesting that the geometry learned under full-view attention also benefits rendering. We will make this scope clearer in the main text and add a discussion of conventional pose-required and pose-free GS methods to the related work.

(Q1) Extension to non-pixel-aligned representations

Our full-view strategy relies on pixel-aligned Gaussians. Full-view attention is safe only because a target view is never rendered from its own prediction, which requires knowing which view each Gaussian comes from. With pixel-aligned Gaussians, we simply drop those predicted from target pixels and render targets from context-view Gaussians. In non-pixel-aligned methods such as C3G and TokenGS, Gaussians are decoded from queries or tokens that aggregate all views, so target information cannot be separated from the scene. Applying full-view attention directly would then let targets be reconstructed from their own content, a shortcut the rendering loss cannot detect.

We also tried predicting Gaussians freely in 3D space from latent tokens, as in non-pixel-aligned methods, while restricting these tokens to attend only to context views. This did not train well in our self-supervised setting. We believe this is because pixel-aligned Gaussians are tied to the predicted camera rays. Their positions are defined by depth along each ray, so the rendering loss also supervises the cameras through the geometry. Free-space Gaussians lose this coupling, and extending our strategy to them remains an open direction that we will discuss in the revision.

Following up on our response to (Q1), we ran an additional experiment that relaxes per-pixel alignment.

Following ReSplat and iLRM, the model predicts one Gaussian per 2×2 pixel cell instead of one per pixel (4× fewer Gaussians), with a learned offset that moves its ray within the cell. Everything else is unchanged. Since each Gaussian is still predicted from a single view, target-view Gaussians can be dropped exactly as before, and full-view attention remains safe. Results on DL3DV, averaged over 5/7/9/11 views:

GaussiansR3R5R10A30
Per pixel (ours)66.681.791.788.0
Per 2×2 cell + offset64.982.392.188.3

R3/R5/R10: RPA@3/@5/@10; A30: AUC@30.

Pose accuracy remains comparable. This refines the requirement stated in our response. The strategy does not need strict per-pixel alignment, but each Gaussian must (i) come from a known source view and (ii) be placed along a ray of that view. Free-space Gaussians decoded from tokens that aggregate all views satisfy neither condition, which is why they remain an open direction. We will include this experiment in the revision.

Reviewer_ZdUuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to learn 3D representations from multi-view unposed images in a self-supervised manner, and treats NVS loss as a training signal. Earlier works avoid trivial solutions by blocking the connection between context and target views, which is called an asymmetry. Instead, they let all images attend to each other and enforce the separation only when rendering the Gaussian scene, which comes from the context views alone. They enforce two auxiliary losses. One distills knowledge from dense views to sparse views. The other matches the features to a frozen pretrained image backbone for semantic knowledge. The method demonstrates robustness on in-domain and out-of-domain testbeds, especially in monocular depth.

Critical strengths
  1. The insight is well-motivated, and the solution is elegant. The proposed asymmetry design is a minimal change that fits existing feed-forward architectures. In a controlled experiment (Fig. 3), where the backbone and training setup are fixed and only the placement of the asymmetry changes, the new design outperforms previous designs on both depth and pose. The resulting model is also compact, with only 183M parameters.
  2. The paper shows that this self-supervised training can serve as an effective pretraining stage. Tab. 4 shows that SSL pretraining followed by supervised fine-tuning beats supervised training alone, with fewer total iterations.
Critical weaknesses
  1. The semantic guidance loss is motivated by textureless walls, repeated patterns, and thin structures. However, the paper doesn’t provide any experiments or visualizations on these regions to show that the loss is helpful. Also, the same Pixio model is used to initialize the encoder and to provide the guidance signal. The loss might therefore mainly keep the encoder close to its initialization, instead of injecting useful semantic knowledge. I think more analysis needs to be done here.
  2. It’s unclear whether this training is truly scalable and generalizable. All experiments use the same feed-forward backbone. Can it be extended to other models? Also, there’s no study of data and model sizes or of mixed-data training. In addition, cross-dataset transfer seems weak, as with the baselines: when trained on RE10K, the performance on DL3DV is much lower than when trained on DL3DV itself. Without these careful studies, the advantage over supervised feed-forward models remains questionable.
Questions for authors
  1. For the semantic guidance loss, could the authors show depth results or visualizations on textureless, repeated, or thin-structure regions, with and without the loss? In addition, could they use a guidance model different from the initialization (e.g., Pixio initialization with a DINOv3 teacher)?
  2. Could the authors provide evidence that the training is scalable and generalizable, even at a small scale? Examples include applying the training to another feed-forward architecture or training on large-scale in-the-wild natural images.
4 comments

We thank the reviewer for the positive assessment and constructive questions. We address each point below. all new results will be added to the revision.

(W1 & Q1) Effect of semantic guidance on challenging regions, and whether it only anchors the encoder

Challenging regions. Textureless walls, repeated patterns, and thin structures give the rendering loss little depth signal, which motivates semantic guidance (Sec. 3.3). Qualitatively, we observe that guidance yields more consistent depth on textureless walls and sharper boundaries around structures. We will add this comparison to the revision.

Guidance from a different teacher. We keep the Pixio initialization and replace the frozen teacher with DINOv3 or DA3, aligned through an MLP projector since their feature spaces differ from the encoder's. With the same projector, Pixio → Pixio gives 81.6 RPA@3 (Tab. 7), close to the default without it. We report camera pose and depth accuracy (δ<1.25 against MoGe-2; MV/SV: multi-/single-view) on DL3DV, averaged over 5/7/9/11 views:

Init → TeacherRPA@3RPA@5RPA@10AUC@30MV δ<1.25SV δ<1.25
Pixio → none (baseline)66.681.791.788.00.7580.638
Pixio → Pixio (ours)82.390.896.093.00.7900.757
Pixio → DINOv377.688.395.191.70.7820.755
Pixio → DA3 †75.487.694.691.30.7800.754

† DA3 was pretrained with 3D supervision.

A teacher different from the initialization also improves both pose and depth over the baseline. Since this alignment pulls the encoder toward a different feature space, the gain cannot be explained by keeping the encoder near its initialization.

Where guidance is applied. We also vary the student layer aligned to the Pixio teacher:

Guided student layerRPA@3RPA@5RPA@10AUC@30
Pixio block 579.590.195.892.4
Pixio block 880.590.595.792.6
Pixio output (default)82.390.896.093.0
Transformer block 3 (view-wise attn.)81.090.496.092.7
Transformer block 6 (global attn.)83.291.696.693.4

Guiding the transformer, which is randomly initialized and never held Pixio weights, gives gains comparable to guiding Pixio itself. This again shows that the loss does more than keep the encoder near its initialization.

Overall, the guidance provides useful information beyond anchoring. Using the same model for initialization and guidance works best, likely because the teacher's features are already compatible with the encoder.

(W2 & Q2) Scalability and generalization (1/2): in-the-wild data

In-the-wild videos and cross-dataset transfer. We add SpatialVID, a large-scale set of in-the-wild monocular videos with diverse camera motion and moving content, to DL3DV training, keeping the architecture, objective, and schedule unchanged. We evaluate on three disjoint sets of 100 held-out SpatialVID scenes grouped by the amount of dynamic content (10 views over a 32-frame span; 48 frames for DL3DV):

R5 / A30DL3DVSpatialVID dyn_lowdyn_highcrowded
DL3DV only80.7 / 87.136.9 / 70.025.4 / 55.724.1 / 55.1
DL3DV + SpatialVID79.1 / 85.874.7 / 86.073.9 / 84.087.4 / 89.2

R5: RPA@5; A30: AUC@30.

As the reviewer notes, a model trained on a single dataset transfers poorly to a new domain. Adding unlabeled in-the-wild videos largely closes this gap, even on near-static scenes, while DL3DV performance remains comparable. Without architectural changes, the model handles moving objects by assigning low opacity to their Gaussians.

We also evaluate both models zero-shot on ORBIT, an unseen benchmark of perspective views rendered from 360° YouTube videos with ground-truth poses, often containing moving people. We sample 10 views from each non-overlapping window of S frames:

STraining dataR5R15A30
64DL3DV only13.648.142.5
DL3DV + SpatialVID17.355.947.9
96DL3DV only12.546.240.3
DL3DV + SpatialVID16.451.944.7

SpatialVID improves accuracy at both spans and every threshold, although the large rotations in ORBIT remain challenging for both models.

Mixed-data training. We also train on six datasets (~114K scenes): DL3DV, RealEstate10K, ACID, ARKitScenes, ScanNet++, and WildRGB-D. This run uses a single GPU for 300K iterations over ~10× more scenes than the main experiments, so its numbers are not directly comparable to Tab. 1–2. As a reference, we train the supervised model (SL) with ground-truth poses on the same data, architecture, and schedule. Both use 10 input views:

R5 / A30RE10K*WildRGB-D*DL3DV*ScanNetv2NRGBD7-ScenesETH3D
SL (GT poses)33.3 / 69.078.4 / 86.429.0 / 68.34.0 / 31.07.9 / 52.24.3 / 33.81.0 / 8.5
Ours (no labels)56.3 / 78.089.1 / 91.844.3 / 63.84.9 / 22.27.2 / 40.25.4 / 26.41.9 / 6.3

* Test split of a dataset in the training mixture.

Ours achieves higher R5 on six of seven datasets, while SL achieves higher A30 on five, mostly out of domain. This matches the trade-off in controlled settings: rendering supervision favors precision, while pose labels reduce large failures.


(W2 & Q2) Scalability and generalization (2/2): other models

Other models. Our backbone follows the alternating view-wise and global attention design of recent feed-forward models (e.g., VGGT, DA3), and the full-view asymmetry applies to any model that predicts cameras and Gaussians from multiple views. We test two changes on DL3DV, averaged over 5/7/9/11 views.

First, we switch both outputs to the common feed-forward 3D design (e.g., VGGT): per-view instead of pairwise camera prediction, and unbounded depth (exp) instead of bounded depth (sigmoid scaled to [0, 8]). Both variants are trained without context alignment and semantic guidance:

OutputsR3R5R10A30MV AbsRel ↓MV δ<1.25SV δ<1.25
Ours (pairwise camera, bounded depth)66.681.791.788.00.1950.7580.638
FF3D-style (per-view camera, unbounded depth)67.582.892.989.00.1980.7650.679

Second, we replace the Pixio encoder with encoders of different pretraining and tokenization, with semantic guidance using each model:

EncoderPretrainingTokenizationResolutionR3R5R10A30
PixioMAE (images)per image256×25682.390.896.093.0
DINOv3self-distillationper image256×25662.479.391.287.6
VideoMAE v2MAE (videos)2-frame tube224×22455.972.984.681.1
DA33D-supervisedper image224×22488.995.898.995.7

Training remains stable with both output designs and all encoders, including VideoMAE v2, which tokenizes two consecutive frames jointly. Our training is thus not tied to a specific model design.

Relation to supervised models. Our training also serves as effective pretraining for supervised models: as shown in Tab. 4, self-supervised pretraining followed by supervised fine-tuning outperforms supervised training alone with fewer total iterations.

Thank you for the detailed responses. I will maintain my positive score.

Thank you for your time and positive assessment. We appreciate your constructive feedback, which has helped us improve the paper, and we will incorporate the discussed changes into the revision.

Reviewer_m4SwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on multi-view self-supervised learning using photometric rendering as training signal, with explicit 3D geometry (i.e., 3D Gaussian) as scene representation.

First, the paper revisits the design space of asymmetry (or information sharing) for context and target views in the self-supervised learning framework. It compares with the strategy of prior works: (1) E-RayZer: It separates the module for camera pose estimation and scene reconstruction. The pose estimation module allows bidirectional attention between context and target views, while scene reconstruction module excludes target-view features to prevent information leakage. (2) NAS3R: A single backbone learns features for both pose estimation and scene reconstruction, but the attention mask prevents target features from seeing context features. Surprisingly, the authors find that a single backbone without any attention mask is sufficient for superior performance for predicting geometry. The key is that target-view rendering should not use target-view scene representations, which would otherwise create shortcut solutions. This strategy allows full bidirectional information sharing among all views and achieves the best results in terms of pose estimation and depth estimation.

Second, this paper shows that two strategies can further improve performance: (1) Self-distillation that distills features from dense views to sparse views; (2) Feature alignment from a pre-trained backbone, similar to how REPA works.

Extensive experiments show that the proposed method outperforms the state-of-the-art multi-view self-supervised learning framework (e.g., NAS3R, E-RayZer), and the proposed two strategies bring consistent gains for pose estimation and reconstruction.

Critical strengths

(1) This paper is well written, easy to follow, with nice figures.

(2) The controlled studies around the asymmetry design (information sharing) in multi-view self-supervised learning are well designed, and they present clear comparison, which offer useful insights for feature work.

(3) The proposed method shows state-of-the-art results, when compared with other baselines. The focused ablation studies show the effectiveness of the proposed strategies.

Critical weaknesses

(1) In Fig. 3, while the comparison of three strategies show clear trade-offs in pose and depth estimation, it remains unclear to me how these different information sharing strategies affect representation learning, i.e., how would the performance on the downstream tasks be affacted? A comparison similar to Table. 4 with different model variants presented would be very informative.

Also, in Fig. 3, it feels a bit counterintuitive to me that although the pose estimation in "separated" allows full bidirectional attention as "full-view" does, it works worse in pose estimation. Could the authors explain why? Would a looser metric like RPA@15 explain the reason?

(2) E-RayZer is trained without pre-trained priors. Is there a "more fair" comparison with E-RayZer where the proposed method removes any pre-trained priors?

(3) In Fig. 4, Table 5, and Table 7, I see the individual contribution of context alignment and feature guidance. Do they bring complementary gains? Also, is there an ablation study about which layers to align/guide with what encoders (e.g., DINO, VGGT, MAE)? I'm curious would different encoders bring different sort of gains.

(4) It's a bit surprising to me that in Table 4, SSL and SL work quite similarly well. Could the authors explain more about why and the experiment setups here.

Questions for authors

I have included my questions in "Critical Weaknesses".

5 comments

We thank the reviewer for the careful reading and constructive questions. We address each point below. All new tables and discussions will be added to the revision.

(1) Effect of the information-sharing strategy on downstream performance, and why the separated design underperforms

Downstream performance. We finetuned each Fig. 3 variant (100K self-supervised iterations on DL3DV) for camera pose with ground-truth supervision, following the SSL+SFT recipe and evaluation of Tab. 4 (50K iterations). For the separated variant, we transfer the backbone and camera head and discard the Gaussian transformer, so all fine-tuned models share the same architecture. D/S/N: DL3DV/ScanNetv2/NRGBD; R3/R5/R15: RPA@3/@5/@15; A30: AUC@30.

Init. (→ SFT 50K)D-R3D-R5D-R15D-A30S-R3S-R5S-R15S-A30N-R3N-R5N-R15N-A30
Separated48.273.196.386.71.34.630.530.016.536.880.270.6
Masked-view65.284.898.190.71.35.937.335.420.044.788.477.2
Full-view (ours)73.489.398.692.43.410.144.741.433.362.093.681.9

The ranking of Fig. 3 holds after finetuning: full-view is the best initialization on every metric and dataset, followed by masked-view and then separated. The gap is largest at strict thresholds and out of domain, showing that the information-sharing strategy affects not only the self-supervised predictions but also how well the learned representation transfers to downstream supervised training.

Why separated underperforms. Although its camera branch has full cross-view attention, the separated design learns pose and geometry in two networks. The camera branch receives the rendering loss only through the predicted poses, a few parameters per view, while a separate transformer learns geometry from the context views. Cross-view reasoning is thus learned twice, and the two tasks interact only through the poses. In full-view, both are decoded from one shared representation, so the pose features also receive dense, per-pixel gradients through the Gaussian prediction. This follows the design of geometry foundation models such as DA3, where a shared backbone with task-specific heads encourages strong task interaction and avoids redundant representations.

As the reviewer suggested, looser metrics clarify this (self-supervised models, averaged over 5/7/9/11 views on DL3DV):

RPA@3RPA@5RPA@10RPA@15RPA@30AUC@30
Full-view66.681.791.794.497.288.0
Separated47.268.187.893.397.984.3

The gap shrinks from 19.4 at RPA@3 to 1.3 at RPA@15 and slightly reverses at RPA@30. Separated thus recovers coarse poses as well as full-view but lacks fine accuracy, which is consistent with its limited downstream benefit above.


(2) Comparison with E-RayZer without pretrained priors

Our controlled comparison (Sec. 3.2, Fig. 3) isolates the design choice from the prior. Its separated variant follows E-RayZer but uses the same pretrained encoder, backbone, data, and training schedule as ours, and even has more capacity (an additional 12-layer transformer). Under these matched priors, full-view still clearly outperforms it (self-supervised models, averaged over 5/7/9/11 views on DL3DV):

RPA@3RPA@5RPA@10RPA@15RPA@30AUC@30
Full-view66.681.791.794.497.288.0
Separated (E-RayZer design)47.268.187.893.397.984.3

The gap also persists after supervised fine-tuning (see (1)). The advantage of our design therefore does not come from the prior.

We acknowledge that this comparison is prior-matched rather than prior-free. Tab. 7 reports our model without pretrained initialization (RPA@3 36.2), but this analysis-scale model is not directly comparable to E-RayZer in Tab. 2, which uses full-scale training and a different evaluation protocol. Retraining our design without prior at full scale is beyond our rebuttal compute budget. We will add this comparison in the revision.

(3) Analyses on context alignment and semantic guidance, and choice of layers and encoders

All numbers are on DL3DV, averaged over 5/7/9/11 input views.

Complementary. We stack both signals on the same baseline:

Ctx. align.Sem. guid.RPA@3RPA@5RPA@10AUC@30
66.681.791.788.0
✓69.884.693.589.9
✓82.390.896.093.0
✓✓83.292.496.993.8

Combining both signals gives the best pose at every threshold. Semantic guidance provides most of the pose gain, and context alignment adds a consistent improvement on top, largest with fewer views.

MetricMethod5 views7 views9 views11 views
RPA@5Sem. guid.85.990.793.093.6
+ Ctx. align.89.692.393.894.0
Δ+3.7+1.6+0.8+0.4
RPA@10Sem. guid.93.396.297.097.5
+ Ctx. align.95.197.197.397.9
Δ+1.8+0.9+0.3+0.4

This matches the role of context alignment in transferring dense-context features to sparse inputs.

Which layer to guide. We vary the student layer aligned to the frozen Pixio teacher:

Guided student layerRPA@3RPA@5RPA@10AUC@30
Pixio block 579.590.195.892.4
Pixio block 880.590.595.792.6
Pixio output (default)82.390.896.093.0
Transformer block 3 (view-wise attn.)81.090.496.092.7
Transformer block 6 (global attn.)83.291.696.693.4

All placements substantially improve over no guidance (66.6 RPA@3). Early Pixio layers are the weakest, and guiding after global (cross-view) attention works best. We will include this ablation in the revision.

Which encoder. The same model initializes the encoder and serves as the frozen teacher:

EncoderRPA@3RPA@5RPA@10AUC@30
Pixio (MAE)82.390.896.093.0
DINOv362.479.391.287.6
VideoMAE v2 †55.972.984.681.1
DA3 (3D-supervised) †‡88.995.898.995.7

† Trained and evaluated at 224×224 (native patch grid), so only approximately comparable. ‡ DA3 was pretrained with 3D supervision; our training remains self-supervised.

We also tested other encoders. Among self-supervised encoders, Pixio gives the best pose, so we use it throughout. For a 3D-supervised encoder, we use DA3 rather than VGGT, since VGGT is only available at ~1B parameters, whereas DA3 offers a base-size model matching ours. DA3, pretrained with 3D supervision, gives the highest pose accuracy. we include it as a reference.

(4) Why SSL and SL perform similarly in Tab. 4, and the experimental setup

Setup. SSL and SL share the same architecture (Pixio ViT-B/16 encoder, 12-layer backbone, camera head), DL3DV training scenes, view sampling, and optimizer. Only the training signal differs:

  • SSL: rendering loss only, no pose labels, 100K iterations.
  • SL: ground-truth relative-pose loss over all view pairs, 200K iterations.

Analysis. SSL and SL are close on AUC@30 and at loose thresholds but differ at strict thresholds. To understand this, we split the DL3DV view pairs of Tab. 4 (pooled over 5–11 views) by ground-truth relative rotation:

PairsModelRPA@3RPA@10RPA@15Failure rate (%)Translation error (°)
Easy (<30°)SSL77.196.698.10.71.5
SL54.693.597.01.02.6
Hard (≥30°)SSL50.986.290.25.51.6
SL44.391.597.20.32.3

Failure rate: pairs with relative pose error above 30°. Translation error: median angular error of the translation direction.

  • SSL is more precise. Rendering penalizes even small pose errors through pixel misalignment, so SSL is more accurate at RPA@3 on easy pairs, with lower translation error.
  • SL is more robust. Under large rotations, view overlap shrinks and the rendering signal weakens, so SSL fails more often on hard pairs, and its gap to SL widens at looser thresholds. The same holds with sparse inputs (5 views: 5.4% for SSL vs. 1.0% for SL) and on NRGBD out of domain (≥30° pairs: 12.3% for SSL vs. 2.2% for SL).

The two balance out on aggregate metrics, which is why SSL and SL appear similar in Tab. 4. Their strengths are complementary, and SSL+SFT, which combines both training signals, performs best on every metric. We will clarify this in the revision.

Additional result for (W2): controlled comparison without pre-trained priors

Following up on our response to (W2), we repeated the controlled comparison of Sec. 3.2 with the pre-trained prior removed from all variants. Every encoder is randomly initialized, and the data, view sampling, optimization settings and 100K iterations are identical, so the only difference is the architecture.

Variant (random init)RPA@5RPA@10AUC@30Depth AbsRel ↓Depth δ<1.25
Separate branch (E-RayZer-style)8.433.445.80.3710.447
Masked attention16.051.756.90.3860.431
Full view (ours)56.677.375.70.2470.652

(DL3DV, mean over 5/7/9/11 views)

Without priors, the gap between designs becomes larger within the same training budget. The separate-branch and masked-attention variants recover only coarse camera pose, while full-view training already learns meaningful pose and depth. We note that longer training might narrow this gap, but within the same budget full-view training learns much faster without a prior. This suggests that the advantage of our design does not rely solely on the pre-trained prior. We will add this comparison to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.