acceptodds
Withdrawn from ICLR 2027

GLARE: GENERALIZABLE LENS-AGNOSTIC RELATIVE-RAY ENCODING FOR 3D RECONSTRUCTION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward 3D reconstruction models inherit strong geometric priors from perspective imagery, but these priors are tied to how image coordinates parameterize viewing directions. When the camera projection changes, the same image-grid relation no longer corresponds to the same observation geometry, limiting the transfer of pretrained multi-view reasoning across lens configurations. We introduce GLARE (Generalizable Lens-Agnostic Relative-ray Encoding), which formulates projection adaptation as conditioning latent multi-view interactions on image-inferred camera-local viewing geometry. For each image, we estimate a continuous field of viewing directions and derive local ray descriptors that characterize how visual tokens sample the camera’s projection. Rather than treating these rays as scene points or imposing them as output constraints, we use their relative descriptor phases to modulate feature comparison and aggregation across views. This separates observation geometry, which describes how each image samples the scene, from scene geometry and camera motion, which remain jointly inferred from visual evidence. The resulting formulation enables a perspective-pretrained reconstruction prior to reason over diverse central-camera projections without requiring calibration, camera poses, or camera-family labels at inference. We therefore shift lens adaptation from camera-specific output modeling to projection-aware latent reasoning, providing a unified mechanism for reusing pretrained multi-view geometric priors across heterogeneous camera systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Relative Ray Encoding, 3D Reconstruction, Calibration-Free, Pose Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_hGSbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GLARE adapts a pretrained π3 model for RGB-only multi-view reconstruction across perspective, fisheye, and equirectangular camera projections. An image-conditioned ray-field estimator predicts continuous camera-local viewing directions. These directions define orthogonal feature transformations that condition attention scores and value transport in a parallel adapter. The image encoder and native aggregator remain frozen, while the ray estimator, adapters, and point/pose branches are trained. The paper evaluates 3D reconstruction, radial depth, and relative camera pose, including zero-shot evaluation on Stanford2D3D and FIORD, and reports improvements over VGGT, π3, fine-tuned π3, and Wid3R on many of the evaluated settings.

Critical strengths
  1. A useful inference setting with a clear practical motivation. Adapting a pretrained multi-view model to different camera projections without requiring calibration, poses, or camera-family labels at inference addresses an important deployment constraint. Predicting observation directions from images and introducing them into feature interactions offers a plausible way to reuse pretrained geometric representations across camera domains.

  2. Substantial improvements on several benchmarks. The reported results show meaningful gains in depth, reconstruction, and pose estimation. For example, on Realsee Real, completion error decreases from Wid3R's 0.185 to 0.073 and depth AbsRel decreases from 0.102 to 0.057. GLARE achieves the lowest reported depth AbsRel and RMSE across all five depth benchmarks. These results support the practical potential of the approach, even though the gains are not uniform across all tasks and held-out datasets.

Critical weaknesses
  1. The independent contribution is not isolated from existing designs and other adaptation changes. Section 3.2 explicitly follows UniK3D's spherical-harmonic ray parameterization, while Section 3.3 adopts a parallel conditioning strategy inspired by UCPE. Transforming Q/K/V and applying an output back-transform also have precedent in UCPE's attention formulation. GLARE's specific orthogonal action built from image-inferred rays may provide a useful extension for reconstruction, but its distinct contribution is not established experimentally. There are no component ablations comparing the proposed adapter with a comparable adapter without ray conditioning, or isolating the benefit of value transport beyond conditioning attention scores. In addition, the method changes the point output to unrestricted XYZ and jointly trains the point/pose branches. Without a clearly matched π3∗ control, the gains cannot be cleanly attributed to the proposed relative-ray mechanism. Appendix F describes GLARE's own optimization in detail, but the trainable modules, data, objectives, and schedule used for π3∗ are not specified sufficiently to assess this comparison.

  2. The evidence for broad camera generalization and the evaluation protocol remain incomplete. Table A2 identifies only two zero-shot datasets: Stanford2D3D with ERP images and FIORD with fisheye images. Both projection families are represented in training. These tests provide evidence of cross-dataset transfer, but do not by themselves establish transfer to unseen projection families or camera-parameter ranges. The held-out pose results are also mixed: on FIORD, GLARE is the best evaluated method but reaches only 6.73% AUC@5°; on Stanford2D3D, its AUC@5° is 48.94%, below π3∗ at 59.70% (Table 3). FIORD contributes only pose evaluation because dense geometric ground truth is unavailable in the current interface, further limiting the evidence for held-out reconstruction performance.

    Several details needed to interpret and reproduce these comparisons remain unspecified, including evaluation split sizes and test-view counts and selection. Test-time scale handling and alignment for reconstruction and depth metrics are also unclear. The extensive descriptions of training normalization and supervision do not resolve these testing questions. Finally, the comparison omits camera-general monocular methods such as UniK3D on the overlapping radial-depth and local-geometry tasks. Such a comparison, with the input-view and alignment differences made explicit, would help establish what the proposed multi-view adaptation adds beyond existing camera-general single-image predictions.

Questions for authors
  1. Methodological contribution and ablations. Which component is the main advance beyond the UniK3D-inspired ray estimator and UCPE-inspired adapter? Can ablations compare a similarly sized adapter without ray conditioning, ray conditioning of attention scores alone, and the full score-and-value conditioning mechanism while keeping the remaining training setup fixed?

  2. π3∗ and evaluation details. Which modules, objectives, training data, and optimization schedule are used for π3∗? Does it receive the same unrestricted XYZ output adaptation? Please also specify evaluation split sizes, test-view sampling, and any scale or rigid alignment used for the reconstruction and radial-depth metrics.

  3. Generalization limits. What explains the low absolute pose AUC on FIORD and the regression relative to π3∗ on Stanford2D3D? Is there evidence separating transfer to new datasets from transfer to camera models or parameter ranges absent from training?

  4. Camera-general monocular comparison. Can UniK3D be evaluated on the shared radial-depth and camera-local geometry tasks with consistent preprocessing and alignment conventions? How much does GLARE improve over this baseline when the difference between single-view and multi-view inputs is made explicit?

Reviewer_YbqNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses a practical problem in 3D reconstruction: models trained primarily on conventional perspective images often struggle to maintain their performance on fisheye or panoramic images. A key reason is that different lenses form images differently: even pixels at the same image location can correspond to entirely different viewing directions. GLARE introduces this directional information into the model to help it understand geometric relationships across different image formation models. The method builds on the pretrained π³ 3D reconstruction model and consists of three main steps:

  1. Estimate viewing rays from images. A small network predicts the ray direction corresponding to each image patch in the camera’s local coordinate system.
  2. Incorporate ray information into attention. Adapters apply ray-dependent feature transformations, influencing both how features are matched and how information is passed across views.
  3. Jointly predict 3D points and camera poses. During training, the original image encoder and feature aggregation backbone are frozen, while only the adapters, ray estimation network, and output heads are updated. At inference time, the method requires no additional camera calibration parameters, camera poses, or lens-type labels. It is important to clarify that this convenience applies only to inference; training still relies on calibrated rays, geometric information, and pose supervision.
Critical strengths
  • It addresses a practically relevant problem. Viewing directions provide a common geometric representation across different image projections, helping a single reconstruction model accommodate multiple lens types.
  • It makes effective use of an existing pretrained model. Adapting an existing backbone is an attractive approach, although the actual computational savings still need to be measured.
  • The reported performance gains are substantial. On Realsee Real, depth absolute relative error (AbsRel) decreases from 0.102 to 0.057 compared with Wid3R, a reduction of approximately 44%, while pose AUC@5° increases from 17.41 to 65.11. GLARE also reports the lowest depth AbsRel and RMSE across all five dense reconstruction benchmarks (Tables 2–3).
Critical weaknesses
  • The main concern is the lack of ablation studies. The paper does not separately evaluate the contributions of predicted rays, the chosen encoding scheme, or transformations applied to the attention value vectors. The prediction heads and training objectives may also affect the final results, so it remains unclear how much of the improvement can be attributed to the proposed adapters.
  • The evidence for generalization remains insufficient. Although the evaluation includes datasets not used during training, their main projection types—fisheye and panoramic—are already represented in the training data. The results therefore provide stronger evidence of generalization to new datasets than of the ability to handle entirely unseen lens types or distortion patterns.
  • The performance advantage does not hold across all metrics. On Stanford2D3D, a dataset not used during training, GLARE achieves a pose AUC@5° of 48.94, below the 59.70 achieved by fine-tuned π³. On ASE, its reconstruction completeness error is 0.247, also higher than the 0.080 achieved by fine-tuned π³ (Tables 1 and 3).
  • The comparability of the experiments and the evidence for efficiency need further support. The paper provides insufficient detail about the training settings of the fine-tuned baselines, making it difficult to determine whether the comparisons strictly control the relevant variables. It also reports no measurements of runtime, memory or GPU memory usage, or trainable parameter counts, leaving the description of the method as “lightweight” without direct empirical support.
Questions for authors
  1. How are scale and alignment handled during evaluation?

  2. Does the comparison with fine-tuned π³ adequately control for other variables?

  3. How accurate are the predicted rays, and how does their accuracy relate to reconstruction gains?

  4. What level of lens generalization do the experiments actually demonstrate?

  5. How can the performance regressions on some metrics be explained?

Reviewer_dVdqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a method to make feed-forward reconstruction models handle a variety of camera models and lens types without the need of retraining. To separate the observed image content from the geometry, a continuous field of view directions is estimated for each image and associated local ray descriptors act as camera model independent tokens for 3D reconstruction. This abstraction allows to use pre-trained feed-forward 3D reconstruction models do deal with a variety of camera models without retraining.

Critical strengths
  • The paper proposes a moderately simple modification to learned feed-forward 3D reconstruction models in order to make them agnostic to changes of the camera lens or the camera model. The core contribution is a "ray field estimator" which generalizes the network-internal matching from simple pixel correspondences to ray-based matching via camera-local viewing directions.

  • The paper is generally well written, the idea clearly presented and design decisions are well motivated and experimentally supported.

Critical weaknesses
  1. The idea and problem setting of making feed-forward 3D reconstruction techniques better generalize across different camera models is not new, e.g. CAM3R tackles the same problem and seems to be the closest and most recent competitor. From this perspective, I am wondering why CAM3R was not selected as a baseline for comparison. Apart from Wid3R none of the selected baselines aims for camera agnostic reconstruction.

  2. Although the paper writing is generally good several parts in the are unnecessarily concise. Given that the paper is more than one full page under the page limit, I feel that some paragraphs would benefit from details, e.g. more elaborate discussion of experimental results. A discussion of limitations would also be very interesting.

  3. Minor issues:

  • L274: "RealseeDB" -> "RealSee3D" ?

  • Although references take a lot of space in this template, it would be nicer to have datasets and methods properly referenced when mentioned first (L274, L277). The references for FIORD and RealSee3D datasets seem completely missing.

Questions for authors
  1. What is the time / compute overhead of the proposed model vs. the standard baseline?

  2. Can the model also handle less common camera models like catadioptric cameras or push-broom cameras?

  3. Can the model handle a different camera model for every frame? The problem formulation (Sec 3.1) and method section seems to allow varying camera models for every frame. If this is the case, it would be nice to include such a case into the evaluation (e.g. on a video with zoom changes), since the selected evaluation datasets have mostly constant camera models and intrinsics.

  4. A discussion of limitations would be useful.

Reviewer_wZgwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Feed-forward 3D reconstruction models like VGGT and π3\pi^3 are trained mostly on perspective images and degrade on fisheye and panoramic images. To address this, GLARE adapts pretrained π3\pi^3 to these cameras using only RGB images as inputs; with no intrinsics (camera labels) or extrinsics during test time. It has two main parts:

  • A ray estimator (adopted from UniK3D) predicts a ray direction for each image patch.
  • An attention adapter runs in parallel with frozen global-attention layers of π3\pi^3. It uses predicted rays to rotate part of the query, key and value features, similar to how RoPE uses 2D positions. This lets cross-view attention use each patch’s viewing direction instead of only its pixel position. The design follows UCPE, a camera-control method for video generation, but uses predicted instead of known rays.

Pointmap and pose heads borrowed from π3\pi^3 are also fine-tuned. The model is trained on ScanNet++, ASE and Realsee and evaluated on these datasets along with zero-shot on Stanford2D3D and FIORD and reports large gains, especially on distorted data.

Critical strengths
  • The problem matters. Since fisheye and panoramic cameras are increasingly common in robotics, autonomy and AR/VR, this is a practical problem. GLARE needs only RGB images at test time, unlike Wid3R which requires an explicit camera-model token (Sec. 2.1). This makes it easier to use with unknown cameras.

  • Large gains on non-perspective datasets. For example, pose AUC@5° on ASE is 40.08 vs 0.59 for π3\pi^3*. GLARE gives the best pose on five of six benchmarks and best normal consistency on all five reconstruction benchmarks (Tables 1, 3).

Critical weaknesses
  1. Main novelty is not isolated by ablations. The ray estimator follows UniK3D’s spherical-harmonic design (Sec. 3.2) and the parallel adapter is inspired by UCPE’s conditioning for video generation models (Sec. 3.3), so the key new step is conditioning attention on predicted rays. But, there are no ablations in the paper or appendix, so it is unclear how much gain comes from this step. The paper also changes the point head to allow points behind the camera (Sec. 3.4), which by itself may help on panoramic and fisheye data? How does the same model perform without the adapter? What happens with an adapter without ray channels, to control for the extra capacity? And how does soft ray conditioning compare with a hard ray constraint on the points?

  2. Most comparisons mix in-domain and zero-shot results. Headline gains (60.7% lower completion error) come from Realsee (in-domain) and are measured against Wid3R rather than π3\pi^3* (since it’s finetuned on these in-domain datasets). On the held-out Stanford2D3D, the gain over π3\pi^3* mostly disappears (accuracy 0.288 vs 0.292, completion 0.244 vs 0.245) and pose is lower (AUC@5° 48.94 for GLARE vs 59.70 for π3\pi^3*). This makes the “consistently state-of-the-art” claim in abstract stronger than evidence supports.

Questions for authors

Clarifications:

  1. How was $\pi^3$* fine-tuned? Which parameters were trained? Does it use the same point head as GLARE (which allows points behind the camera)?

  2. Wid3R appears not to be fine-tuned on the same training data. Was this considered? Without it, π3\pi^3* is the only baseline trained on the same data.

  3. Which camera-model token was Wid3R given on each dataset? Its pose AUC@5° is 0.00 on both ASE and FIORD (Table 3).

  4. What is the performance of the ray estimator itself on each dataset, especially the zero-shot ones?

  5. Which π3\pi^3 blocks have an adapter? Fig. 1 shows only blocks 1 and 17. How many parameters does the adapter add and how much slower is it than π3\pi^3?

  6. Training uses 2–24 views (Sec. 4.1). How does GLARE perform with a single image (monocular setting)?

Suggestions:

  1. Fig. 2 is hard to read as evidence. The rows have no dataset or camera labels, so it’s hard to tell which samples are in-domain and which are zero shot. Labelling the rows, highlighting regions to compare pointclouds and adding π3\pi^3* visualizations would help.

  2. More qualitative results would help. There is only one unlabeled qualitative figure in the entire paper. A few more examples, including in-the-wild, might support the claims.

  3. A justification/reasoning description would help explain the pose results. Pose AUC@5° is low on FIORD (6.73) and ASE (40.08), while it is above 50 on other datasets (Table 3). Both are fisheye datasets, so a short discussion of what makes fisheye pose estimation harder for GLARE would be useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.