acceptodds
Under review as a conference paper at ICLR 2027

UniReg: Joint Multi-Source Learning for Universal Image-to-Point-Cloud Registration

Abstract

Image-to-point-cloud registration recovers the 6-DoF pose of a query RGB image relative to a reference point cloud. While recent learning-based methods perform well on individual benchmarks, they remain largely tied to a fixed sensing process: outdoor sparse LiDAR and indoor RGB-D reconstructions differ substantially in density, scanning pattern, metric scale, completeness, and image–map overlap. As a result, models trained under one acquisition setup transfer poorly to another. We analyze this failure mode and attribute it to representations that entangle acquisition cues with scene geometry, pose pathways that are sensitive to density and scale, and the limited effectiveness of naively mixing heterogeneous datasets. We present UniReg, a single feed-forward network for I2P registration across indoor and outdoor scenes, LiDAR and RGB-D inputs, and fragment- to scene-level geometry. UniReg separates acquisition context from geometric reasoning through token-space conditioning, predicts sensor-robust ray-centric multi-geometry, and aggregates these predictions via confidence-aware fusion. To support this factorization, we introduce a scene-level ScanNet I2P dataset and a CARLA corpus with diverse camera–LiDAR configurations, and train jointly with standard benchmarks. With one set of weights, UniReg matches or surpasses specialist models across established indoor and outdoor evaluations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-to-Point Cloud Registration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oA39OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents UniReg, a feed-forward network for image-to-point-cloud (I2P) registration across different sensing setups, spanning outdoor LiDAR, indoor RGB-D fragments, and scene-level reconstructions.

The framework is built on three core designs:

  • Token-space conditioning: Employs foundation models (DINOv2 for images and PTv3 for point clouds) with a camera token and learnable scene tokens to process cross-modal interleaved attention.

  • Ray-centric multi-geometry prediction: An MLP predicts the initial 6-DoF pose from the camera token, while two DPT heads predict dense depth and a Plücker ray map in the point-cloud frame.

  • Confidence-aware fusion: Uses predicted dense ray and depth confidence maps to update the camera token via cross-attention, regressing the final refined pose.

UniReg is jointly trained on six sources using domain-indexed scene-token banks and coordinate scale factors. The authors report competitive or superior performance compared to per-dataset specialist models across standard benchmarks.

Critical strengths

I found two main strengths in this work:

First, the paper provides a clear and coherent framing of cross-sensor degradation in I2P registration. Instead of treating sensor differences as a domain shift, the authors categorize the underlying bottlenecks into three coupled failure modes: entangled representations, sensor-sensitive geometric pathways, and negative interference in naive joint training. The proposed framework maintains a clean, one-to-one mapping between these identified challenges and its architectural components, making the methodological narrative intuitive and well-structured.

Second, joint ray and depth supervision is conceptually appealing. Table 3 provides useful empirical evidence that the two supervision signals are complementary and using both yields the lowest reported translation and rotation errors across the evaluated datasets.

Critical weaknesses

Despite the clear framing, several major issues weaken the paper's core claims and empirical reliability:

First, the baseline numbers and evaluation protocols appear misaligned. While comparing this work with recent I2P paper (e.g., Table 5 of arXiv:2603.26262), the baseline entries in Table 2 (formatted as "A ± B", such as 3.026±2.649∘3.026 \pm 2.649^\circ for 2D3D-MATR) is the same as the Mean and Median values reported in that work. The manuscript never defines what "±" signifies. If "±" connects the mean and the median, this is non-standard and misleading. It is also unclear whether UniReg’s own entries use the same convention, mean ± standard deviation across test pairs, or variability across training runs. Furthermore, the documented evaluation setup differs from prior work, including the 30% overlap threshold on 7-Scenes (compared to the standard 50% cutoff in baselines such as CA-I2P), making it unclear whether all entries were evaluated on identical test pairs.

Second, the paper omits a direct comparison with RayI2P (Li et al., 2026), one of the most directly related prior methods. RayI2P already uses ray prediction and ray-guided pose regression to avoid explicit 2D–3D matching. Section 2 mischaracterizes RayI2P as a correspondence method, and Table 1 omits it entirely despite reporting competitive results on the same benchmark datasets (e.g., 0.09 m RTE on KITTI). Without a benchmarked comparison under a common evaluation protocol, the technical progress attributable to UniReg remains unsubstantiated.

Third, cross-sensor generalization and feature disentanglement are overclaimed. The model relies on explicit domain metadata via separate indoor/outdoor token banks and manual coordinate scalings. This does not invalidate the single-checkpoint setting, but the scope of generalization should be stated more precisely. Moreover, while Table 4 confirms scene tokens improve accuracy, it does not prove that sensor-specific factors are truly disentangled from geometry.

Finally, the mechanism of confidence-aware fusion lacks isolated validation. The strong claim that confidence weighting prevents noisy depth from corrupting the pose is not theoretically guaranteed by formulation alone. Controlled ablations such as removing fusion entirely or using uniform weights without confidence are required to substantiate this benefit.

Questions for authors

Could the authors clarify the following points during the rebuttal?

Clarification of Table 2:

What do the values after "±" represent in Table 2? Please clarify whether all methods, including UniReg, follow the same statistical definition and aggregation procedure.

Consistency in Evaluation:

Are the baseline numbers in Table 1 and Table 2 re-evaluated under UniReg's exact test protocol (including 6-DoF pitch/roll perturbations on outdoor sets and the 30% overlap threshold on 7-Scenes), or were they compiled from prior papers? If compiled, could you report baselines evaluated on the exact same test pairs and perturbations?

Relation and Comparison to RayI2P:

Could you clarify the conceptual differences between your ray formulation and RayI2P? Additionally, please include RayI2P's reported benchmark numbers in Table 1 to provide a direct and fair comparison.

Shared vs. Domain-Specific Token Banks:

How does UniReg perform with a single shared scene-token bank, while keeping the current normalization and training settings unchanged? Separately, how sensitive is the model to the domain-dependent scale normalization? To support the broader transfer claim, could you evaluate a sensor configuration or data source excluded from training, or clarify that the current evidence concerns acquisition regimes represented in training?

Ablation of Confidence-Aware Fusion:

Could you compare the proposed fusion against the initial pose from Eq. 5 and against the same fusion architecture with both confidence maps fixed to one (with learned attention weights preserved and ray/depth supervision unchanged)? A comparison of predicted confidence against actual geometric error would also help assess the reliability claim.

I am open to revising my rating if the authors address the concerns raised above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer oA39 (1/2)

Dear Reviewer oA39,

We sincerely thank you for recognizing UniReg's coherent framing and complementary ray/depth design. We also appreciate your precise identification of the evaluation and positioning issues. We address each concern below and indicate where the manuscript has been revised.

W1, Q1 & Q2 — Statistics and evaluation consistency

Meaning of "±". The manuscript omitted its definition and mixed two benchmark-specific conventions. Table 1 follows outdoor I2P work and reports mean ± standard deviation over test pairs. Table 2 follows the indoor convention and reports mean and median; using "±" there was inappropriate. In the revision, Sec. 4.3 now defines "±" as mean ± standard deviation, and Table 2 reports separate Mean and Median columns, as stated in its caption.

7-Scenes overlap. The stated 30% threshold is a typographical error. Our data-generation code uses the standard 50% threshold, official split, and the same image–point-cloud pairs as 2D3D-MATR and CA-I2P. No additional 30%-overlap pairs were evaluated. We have corrected this in Sec. 4.2, Appendix A.1.5, and the caption of Table 2.

Source of baseline numbers. On RGB-D Scenes V2 and 7-Scenes, Table 2 values are compiled from prior papers, which use the same indoor pairs, metrics, and Mean/Median aggregation.

Table 1 values are also quoted from the corresponding papers and were not rerun under our exact perturbations. Most compared methods estimate pose from scratch through correspondences and PnP-RANSAC; our perturbation is not an initialization supplied to their solver. UniReg further uses a broader 6-DoF range, including large pitch and roll (App. A.2.3), so it is not evaluated under an easier pose distribution. We now state in the outdoor results of Sec. 4.4 that these baseline values are reported from the original papers, and we interpret small numerical gaps as competitive rather than as a strict ranking.

W2 & Q3 — Relation and comparison to RayI2P

We agree that RayI2P is the closest ray-based prior. It predicts patch-wise ray bundles and performs ray-guided pose regression without explicit 2D–3D matching. We add a fair benchmark-level comparison using the same datasets, official splits, and metrics (all test pairs retained; Acc.: RTE < 2 m and RRE < 5°):

MethodTrainingKITTI RTE/RRE/Acc.nuScenes RTE/RRE/Acc.
RayI2PSeparate per-dataset models0.09/0.63/99.750.39/1.48/96.61
UniRegOne six-source checkpoint0.10/0.59/99.910.39/0.69/99.54

Our contribution is not the first use of rays, but ray-centric multi-geometry: joint pose, dense depth, and per-pixel cloud-frame Plücker-ray prediction. Rays provide sensor-robust projective geometry, depth supplies metric range, and confidence-aware fusion combines them within one model spanning LiDAR, RGB-D fragments, and complete scenes. In the revision, we discuss RayI2P in the Related Work (Sec. 2), add it to Table 1, and compare against it in the outdoor results of Sec. 4.4.

AuthorsOpenReview ↗

Response to Reviewer oA39 (2/2)

W3 & Q4 — Unified model, token banks, and normalization

Why this remains one model. A single checkpoint and shared registration pathway handle every source: all encoders, eight attention layers, ray/depth heads, fusion, and pose heads are shared. There are no dataset-specific backbones or output heads. The indoor/outdoor factors are reversible preprocessing that place roughly 50 m outdoor and 10 m indoor coordinates/targets in comparable ranges; translations are converted back to meters. Relative to one shared bank, acquisition conditioning adds only 16 tokens (16,384 parameters).

No sensor metadata. A fixed 30 m point-cloud-span threshold selects the normalization and token bank. Thus, routing uses only observed input geometry, not sensor or dataset identity.

Separate sensitivity controls. We isolate token sharing, normalization, and generic token capacity while holding all other settings fixed:

Token designNormalizationKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
No tokensoutdoor /5, indoor /10.23/1.340.123/2.9620.103/2.288
16 shared register tokensoutdoor /5, indoor /10.13/0.740.084/2.1050.058/1.402
Two randomly routed banksoutdoor /5, indoor /10.14/0.770.086/2.1420.060/1.437
One shared banknorm=1 for all0.17/0.860.085/2.1180.059/1.421
Conditioned banksnorm=1 for all0.15/0.790.073/1.8610.048/1.192
Conditioned banksoutdoor /5, indoor /10.10/0.590.071/1.8160.045/1.116

No tokens vs. shared registers measures generic capacity; random vs. conditioned banks isolates acquisition-aligned routing with identical parameters and computation; norm=1 vs. outdoor /5, indoor /1 measures normalization sensitivity. Most importantly, one shared bank with norm=1 still registers every benchmark. Conditioning improves stability and accuracy but is not required for unification. All rows are reported in Table 4 of the revision, with the analysis in Sec. 4.6.

Scope of generalization. The original tables demonstrate one-checkpoint performance across regimes represented during joint training. We now additionally hold out complete CARLA configurations:

Training configurationUnseen test configurationRTE (m) ↓RRE (°) ↓
FoVs 60°, 90°, 120°FoV 75° (interpolation)0.150.73
FoVs 60°, 75°, 90°FoV 120° (extrapolation)0.231.12

On the fully held-out CARLA Town05, the same checkpoint reaches 0.14 m RTE, 0.71° RRE, and 99.32% Acc. Both results are added to Appendix A.3 (Tables 8 and 9) of the revision.

Intrinsics and ray consistency. The network receives only the image and point cloud; KK is used during training to construct ray/depth targets but is not an inference input. Ray directions are unit-normalized, and each training ray field is generated consistently from one physical camera. The held-out-FoV results above show that registration remains accurate on unseen projective geometry without supplying KK at inference.

W4 & Q5 — Confidence-aware fusion

Table 3 establishes complementary ray/depth supervision but does not isolate fusion or confidence. We therefore retain identical dense heads and supervision and vary only fusion:

FusionKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
Initial pose, Eq. (5)0.18/1.020.104/2.5830.078/1.802
Unweighted fusion (Cray=Cdepth=1C_{\rm ray}=C_{\rm depth}=1)0.13/0.760.084/2.1370.058/1.394
Learned-confidence fusion0.10/0.590.071/1.8160.045/1.116

The first difference measures dense multi-geometry fusion; the second isolates learned confidence from the attention architecture. We have added this ablation as Table 5, with its analysis in Sec. 4.6 of the revision.

Thank you again for your detailed and constructive feedback. We hope the corrected evaluation, RayI2P comparison, and controlled experiments resolve your concerns and support a more favorable assessment.

Sincerely,
The Authors

Reviewer_DPM4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a method that address the gap in multiple training data sources in Image-to-Point registration. The paper use a network to regress the pose of a input RGB image by providing a point cloud. Since data from multiple sources differs substantially, the paper prepend several scene tokens to image tokens to disentangle scene related information such as metric scale, point density, overlap, and imaging style. The paper proposed a scene-level ScanNet I2P dataset and a CARLA corpus to support this factorization. The network contains ray and depth heads and is demonstrated to work complementarily. The pose token is refined by interacting with the predicted ray, depth, and confidence. The experiments show SOTA performance compared with methods trained with single source of data.

Critical strengths
  • The problem is clearly identified and well motivated.
  • SOTA performance on multiple benchmarks.
  • The ablation study shows the complementary effect of ray and depth supervision, importance of scene tokens and CARLA training data.
Critical weaknesses
  • There is no garantee that the scene tokens actually help to disentangle the metric scale, point density, overlap, and imaging style from the others. The scene tokens is shown to be unconstrained and there is no loss or evidence to distinguish it from register tokens [1]. So the performance may just benefit from adding register tokens, not the disentanglement.

  • There is no ablation that shows the effect of CONFIDENCE-AWARE FUSION. Although the ablation study show the importance of ray and depth supervision, it mismatch the major claim of the confidence aware fusion. As shown in Figure3, the difference between results before and after is not obvious.

If the above issue is not addressed, the SOTA performance may be due only to data scaling; even if state-of-the-art performance is achieved, its significance to the field would be limited.

Additional concerns:

  • The overall training loss does not include the camera head before confidence fusion.

  • The parameterization of rotation may be not suitable for learning, since rotation is not in Euclidean space. R6D [2] may be a better choice.

[1] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024, May). Vision transformers need registers. In International conference on learning representations (Vol. 2024, pp. 2632-2652).

[2] Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. (2019, June). On the continuity of rotation representations in neural networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5738-5746). IEEE.

Questions for authors
  • How to show that the scene tokens actually help the disentanglement instead of optimization of the model?
6 comments
AuthorsOpenReview ↗

Response to Reviewer DPM4

Dear Reviewer DPM4,

We sincerely thank you for recognizing the importance of the problem, the strong multi-benchmark results, and the complementary ray/depth supervision. We address every concern below and indicate where the manuscript has been revised.

W1 & Q1 — Scene tokens vs. generic registers, and data scaling

Your concern is whether scene tokens encode acquisition context or merely improve optimization like generic registers. The original scene-token ablation alone cannot distinguish these explanations, so we add capacity-controlled comparisons:

Token designKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
No tokens0.23/1.340.123/2.9620.103/2.288
16 shared register tokens0.13/0.740.084/2.1050.058/1.402
Two randomly routed 16-token banks0.14/0.770.086/2.1420.060/1.437
Indoor/outdoor-conditioned banks0.10/0.590.071/1.8160.045/1.116

Shared registers measure generic token capacity. Random and acquisition-conditioned banks have exactly the same parameters, active tokens, and computation; their difference isolates acquisition-aligned routing. We further swap the banks at inference while retaining the correct normalization:

RoutingKITTI RTE/RRE7-Scenes RTE/RRE
Correct bank0.10/0.590.071/1.816
Swapped bank0.21/1.180.109/2.640

The degradation under random routing and bank swapping shows that these tokens carry acquisition-relevant context rather than acting only as generic registers. We have added these controls to Table 4 in Sec. 4.6 of the revision, together with an explicit discussion of generic capacity versus acquisition conditioning.

W2 — Is confidence-aware fusion effective?

Table 3 isolates ray/depth supervision, not fusion. We therefore retain the same dense predictions and supervision and vary only the fusion:

FusionKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
Initial pose, Eq. (5)0.18/1.020.104/2.5830.078/1.802
Unweighted fusion �0��0�0.13/0.760.084/2.1370.058/1.394
Learned-confidence fusion0.10/0.590.071/1.8160.045/1.116

The first difference measures the value of feeding dense geometry back into the camera token; the second isolates learned confidence from the attention architecture. We have added this ablation as Table 5, with its analysis in Sec. 4.6 of the revision.

Additional concern 1 — Is the pre-fusion camera head supervised?

Yes. Equation (10) supervises both the initial pose (R,t)(R,t) and fused pose (R⋆,t⋆)(R^\star,t^\star); its first two terms optimize the pre-fusion camera head directly. Because the camera head and the fused pose share the same loss, the gap between them in Table 5 reflects fusion rather than extra supervision.

Additional concern 2 — Quaternion vs. 6D rotation

We directly supervise normalized quaternions with a consistent sign convention, following VGGT and related pose models (App. A.2.2). Quaternions are compact and yield a valid rotation after normalization, but have a q/−qq/-q sign boundary. The suggested 6D representation avoids this ambiguity and may provide smoother optimization, at the cost of a redundant representation and orthogonalization. Thank you for highlighting this useful trade-off.

Thank you again for your constructive feedback. We hope these controlled experiments resolve the concerns about register tokens, data scaling, confidence fusion, and pose supervision, and support a more favorable assessment.

Sincerely,
The Authors

Reviewer_DPM4OpenReview ↗

Question about the condition banks

Thanks for the author's responce. I have a few questions:

  • Can I interpret this as scene-conditioned training helping the training? If so, what are the scene tokens like at test time, and is it necessary to perform scene classification in advance?
  • Aside from mixing the datasets for training, may I ask what you would consider the main novel contributions of your work?
  • The importance of both indoor and outdoor settings is not highlighted. Are there any applications that require both indoor and outdoor PCR? Also, the comparison does not cover SOTA methods on both indoor and outdoor benchmarks, such as CA-AP2 on KITTI and nuScenes, and RayI2P on ScanNet V2 and 7-Scenes, making readers wonder whether unifying these two tasks is necessary or straightforward given the sensor gap, as prior methods do not evaluate their performance on both tasks.
AuthorsOpenReview ↗

Response to Reviewer DPM4: Test-time Scene Tokens, Contributions, and Indoor–Outdoor Unification

Dear Reviewer DPM4,

Thank you for the follow-up questions.

Q1 — Scene tokens at test time. Conditioning helps joint training, but the tokens are also part of the inference network. They are learned parameters fixed after training. At test time, the selected bank is prepended to the image tokens and updated by the same attention layers, so it conditions the pose and dense heads on each input pair. No scene classifier or label is needed. There are only two banks, and the bank is chosen by a deterministic check on the input cloud: an extent above 30 m uses the outdoor setting. Outdoor (≈50 m) and indoor (≈10 m) extents differ by a wide margin. Routing is also not a prerequisite: with one shared bank and no rescaling (Table 4, row (e)), the same network still registers all benchmarks. Swapping the banks (row (d)) degrades accuracy, confirming that the context is used at inference.

Q2 — Contributions beyond data mixing. All ablations use the identical six-source mixture, so they show what each component adds beyond the data. Our main contributions are:

  • Ray-centric multi-geometry. Per-pixel Plücker rays in the cloud frame do not depend on cloud density, completeness, or extent, which lets one pose pathway serve LiDAR, RGB-D fragments, and full scenes.
  • Confidence-aware fusion of ray and depth evidence into the pose token.
  • Lightweight acquisition conditioning, which adds only 16,384 parameters.
  • A cross-sensor setting with scene-level ScanNet and a sensor-varying CARLA corpus.

Q3 — Why indoor and outdoor, and comparison with SOTA.

  • Applications. Many systems cross the indoor–outdoor boundary within one session: service and delivery robots, AR localization in campuses and stations, vehicles entering garages and tunnels, and drones inspecting buildings. With specialists, these systems need one model per regime plus switching logic; one model removes both.
  • Comparison. The revision compares UniReg with the latest specialist in each regime on its own benchmarks: RayI2P on KITTI and nuScenes (Table 1), and CA-I2P on RGB-D Scenes V2 and 7-Scenes (Table 2). Each specialist is trained separately per dataset and evaluated in one regime only. A single UniReg checkpoint is competitive with or better than both, with consistently lower rotation error.
  • Why unification is not straightforward. Even within RGB-D, changing the reference from fragments to full scenes is hard for specialists (Table 2, ScanNet). Simply training on the mixed data without our components performs much worse.

Sincerely,

The Authors

Reviewer_PYx5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniReg, a unified feed-forward network for heterogeneous image-to-point-cloud (I2P) registration, aiming to estimate 6-DoF camera poses across LiDAR and RGB-D inputs, including local point-cloud fragments and complete scene-level reconstructions, using a single model. To address the limited cross-sensor generalization of existing I2P methods, the authors introduce Token-Space Conditioning with learnable camera and scene tokens. The proposed Ray-Centric Multi-Geometry Prediction module jointly predicts camera poses, rays, and depth, leveraging ray representations to reduce sensitivity to point-cloud sampling density and geometric layout. These geometric predictions are subsequently integrated into the camera token to estimate the final camera pose. UniReg is jointly trained on six data sources, including KITTI, nuScenes, RGB-D Scenes V2, 7-Scenes, CARLA, and ScanNet, and achieves competitive or superior performance compared with dataset-specific specialist models across multiple indoor and outdoor benchmarks.

Critical strengths
  1. The proposed method employs a shared cross-modal feature fusion network that integrates camera pose, ray, and depth prediction into a unified feed-forward architecture, without relying on explicit 2D–3D correspondence estimation or iterative pose optimization.

  2. The authors construct diverse training datasets covering different sensor types, acquisition configurations, and indoor/outdoor environments, providing valuable support for heterogeneous multi-source I2P registration.

  3. Through joint training, UniReg achieves strong registration accuracy with a single model across both outdoor benchmarks, such as KITTI and nuScenes, and indoor benchmarks, including 7-Scenes and ScanNet, matching or outperforming several dataset-specific specialist methods.

Critical weaknesses
  1. The proposed framework does not achieve complete unification across sensing regimes. The model employs separate scene-token banks for indoor and outdoor samples, meaning that the two acquisition regimes do not share the same contextual parameters. Moreover, indoor and outdoor inputs rely on manually predefined scale normalization factors. These domain-specific designs raise concerns about whether the framework truly achieves unified, sensor-agnostic registration.

  2. The effectiveness of Token-Space Conditioning in disentangling acquisition-specific information from geometric representations remains insufficiently validated. One of the key limitations of existing methods identified by the authors is that learned features are highly entangled with sensor acquisition characteristics. However, it is unclear whether introducing scene tokens can effectively disentangle these factors from geometric features. Additional controlled experiments are needed to substantiate this central claim.

  3. The experimental evaluation provides limited evidence of cross-sensor generalization. Although the model is jointly trained on six datasets, it is evaluated primarily on data sources and sensor configurations already represented during training. Therefore, the reported results mainly demonstrate the effectiveness of multi-source joint training rather than generalization to unseen sensing conditions. Additional experiments are needed to support the claimed generalization capability. Furthermore, the paper lacks a direct comparison with the closely related work RayI2P under a consistent evaluation protocol.

Questions for authors
  1. Could the authors provide additional ablation studies on the Confidence-Aware Fusion module and comparisons using different backbone architectures? Furthermore, quantitative results on the held-out CARLA test set are currently missing and would be valuable for evaluating the proposed method.

  2. How are training samples from the six data sources sampled and weighted to ensure balanced joint training? Could the authors construct zero-shot cross-sensor generalization experiments using the existing datasets to further evaluate the model's transferability to unseen acquisition conditions?

  3. Could the authors compare the current domain-specific scene-token banks and scale normalization strategy with a fully shared scene-token design and unified normalization? Additionally, how does the network obtain the camera intrinsic matrix (K)? When the sensor type is unknown at inference time, or when the input exhibits an unseen scale distribution, how does the model determine the appropriate configuration? Finally, how does the network ensure the geometric consistency of predicted ray directions and their angular distribution when processing test images with previously unseen fields of view (FoVs)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PYx5 (1/2)

Dear Reviewer PYx5,

We sincerely thank you for recognizing UniReg's unified feed-forward design, diverse training sources, and strong single-checkpoint performance. We address each concern below.

W1 & Q3 — Is UniReg truly unified and sensor-agnostic?

One shared registration network. One checkpoint handles all six sources without retraining or test-time adaptation. DINOv2, PTv3, all attention layers, ray/depth heads, fusion, and pose heads are shared. The only regime-conditioned parameters are one additional bank of 16 scene tokens (16,384 parameters relative to a shared bank).

Why two normalizations? Outdoor clouds span roughly 50 m, versus about 10 m indoors. Dividing outdoor coordinates, depth, and translation by 5 (and indoor values by 1) simply places inputs and targets in comparable numerical ranges for the same shared heads; predictions are converted back to meters.

Fully shared control. A stricter variant with one token bank and norm=1 for every source still registers all benchmarks:

SettingKITTI RTE/RREnuScenes RTE/RRERGB-D RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
One bank, norm=10.17/0.860.45/0.880.077/2.1470.085/2.1180.059/1.421
Reported UniReg0.10/0.590.39/0.690.069/1.9080.071/1.8160.045/1.116

Thus, conditioning improves optimization at negligible cost but is not required for unification. We have added this comparison to Table 4 (rows (e)–(g)) and discuss it in the scene-token ablation of Sec. 4.6.

Intrinsics and unknown sensor. The forward network receives only the image and point cloud; KK is used during training to construct ray/depth targets but is not an inference input. For metric point clouds, scale is observable from their coordinates. The configuration is selected from the observed point-cloud span using a fixed 30 m threshold, without sensor or dataset metadata.

Unseen FoVs. Ray consistency is learned from dense ray maps generated by one physical camera model per training sample, rather than imposed through KK at inference. The direction is explicitly unit-normalized, while the shared DPT decoder jointly predicts the complete ray field from image–point-cloud interactions. Held-out-FoV results are reported under W3 below.

W2 — Do scene tokens provide acquisition conditioning?

The original scene-token ablation alone cannot exclude the generic benefit of adding register tokens. We therefore keep data, normalization, architecture, and optimization fixed and vary only the tokens:

Token designKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
No tokens0.23/1.340.123/2.9620.103/2.288
16 shared register tokens0.13/0.740.084/2.1050.058/1.402
Two randomly routed 16-token banks0.14/0.770.086/2.1420.060/1.437
Indoor/outdoor-conditioned banks0.10/0.590.071/1.8160.045/1.116

Shared registers measure generic token capacity. Random and acquisition-conditioned banks have identical parameters, active tokens, and computation; their difference isolates acquisition-aligned routing. We additionally swap the banks at inference while retaining the correct normalization:

RoutingKITTI RTE/RRE7-Scenes RTE/RRE
Correct bank0.10/0.590.071/1.816
Swapped bank0.21/1.180.109/2.640

Generic registers and random routing recover only part of the gain, and swapping the banks degrades accuracy markedly. The tokens therefore carry acquisition-relevant context rather than acting only as generic registers. They do not prove identifiable separation of every individual factor; accordingly, we adopt the empirically supported term acquisition conditioning in place of "factor-wise disentanglement". These controls are added to Table 4 (rows (a)–(d)) in Sec. 4.6 of the revision, together with the corresponding analysis.

AuthorsOpenReview ↗

Response to Reviewer PYx5 (2/2)

W3 & Q2 — Unseen configurations, balanced sampling, and RayI2P

Unseen acquisition configurations. We hold out CARLA camera settings entirely during training:

Training configurationUnseen test configurationRTE (m) ↓RRE (°) ↓
FoVs 60°, 90°, 120°FoV 75° (interpolation)0.150.73
FoVs 60°, 75°, 90°FoV 120° (extrapolation)0.231.12

UniReg remains accurate on both unseen fields of view without receiving KK at inference, indicating that the learned ray field transfers to new projective geometry; extrapolation to the wider 120° camera is harder, as expected. This experiment is added to Appendix A.3 (Table 8) of the revision.

Balanced sampling. For each training sample, we first select one of the six sources uniformly with probability 1/61/6, then sample a pair uniformly from that source. Every source therefore has equal expected contribution regardless of dataset size; no source-specific loss weighting is used. This sampling procedure is now stated explicitly in Sec. 4.1.

RayI2P. We agree that RayI2P is the closest ray-based prior. It predicts patch-wise ray bundles and performs ray-guided pose regression without explicit 2D–3D matching. We add a fair benchmark-level comparison using the same datasets, official splits, and metrics (Acc.: RTE < 2 m and RRE < 5°):

MethodTrainingKITTI RTE/RRE/Acc.nuScenes RTE/RRE/Acc.
RayI2PSeparate per-dataset models0.09/0.63/99.750.39/1.48/96.61
UniRegOne six-source checkpoint0.10/0.59/99.910.39/0.69/99.54

Our contribution is not the first use of rays, but ray-centric multi-geometry: joint pose, dense depth, and per-pixel cloud-frame Plücker-ray prediction. Rays provide sensor-robust projective geometry, depth supplies metric range, and confidence-aware fusion combines them within one model spanning LiDAR, RGB-D fragments, and complete scenes. In the revision, we discuss RayI2P in the Related Work (Sec. 2), add it to Table 1, and compare against it in the outdoor results of Sec. 4.4.

Q1 — Fusion, backbone generality, and held-out CARLA

Confidence-aware fusion. We retain identical dense heads and supervision and vary only the fusion:

FusionKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
Initial pose, Eq. (5)0.18/1.020.104/2.5830.078/1.802
Unweighted fusion �0��0�0.13/0.760.084/2.1370.058/1.394
Learned-confidence fusion0.10/0.590.071/1.8160.045/1.116

The first difference measures dense multi-geometry fusion; the second isolates confidence weighting. Both contribute consistently on all three benchmarks. We have added this ablation as Table 5, with its analysis in Sec. 4.6 of the revision.

Backbone generality. We independently replace DINOv2 with ResNet-50 and PTv3 with KPConv, keeping all subsequent layers, losses, data, and optimization fixed:

Image backbonePoint backboneKITTI RTE/RRE7-Scenes RTE/RREScanNet RTE/RRE
ResNet-50KPConv0.21/1.170.108/2.6930.083/1.962
ResNet-50PTv30.16/0.940.093/2.3710.067/1.618
DINOv2-LKPConv0.13/0.750.080/2.0420.056/1.364
DINOv2-LPTv30.10/0.590.071/1.8160.045/1.116

Consistent behavior across CNN/KPConv and foundation backbones shows that the formulation is not tied to one backbone architecture. This experiment is added to Appendix A.3 (Table 7) of the revision.

Held-out CARLA scenes. Town05 is excluded entirely from training:

Test setRTE (m) ↓RRE (°) ↓Acc. (%) ↑
CARLA Town050.140.7199.32

These results provide quantitative evaluation on unseen scenes under the diverse CARLA camera and LiDAR configurations. This result is added to Appendix A.3 (Table 9) of the revision.

Thank you again for your constructive feedback. We hope these clarifications and controlled experiments resolve your concerns and support a more favorable assessment.

Sincerely,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.