acceptodds
Under review as a conference paper at ICLR 2027

CandidateShift: Availability-Aware Learning for Descriptor-Free 2D-3D Matching

Abstract

Descriptor-free 2D-3D matching associates image keypoints with map landmarks without using local visual descriptors. Retrieval and map sampling can remove every valid correspondence for a keypoint, changing its correct matching outcome to unmatched. We present CandidateShift, a training method that separates match availability from conditional landmark identity under candidate deletion. Geometric correspondence labels supervise whether a valid match survives. For keypoints that remain matchable, the prediction on the full candidate set is restricted and renormalized over retained landmarks to provide a soft target for the prediction on the reduced set. Both predictions come from a shared matcher and receive direct correspondence supervision. Compared with the same matcher trained with supervised candidate perturbations, CandidateShift improves conditional identity accuracy under candidate deletion and yields more accurate localization, with gains that persist under stronger outlier contamination.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
2D-3D Matching, Visual Localization, Camera Pose

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SjJDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CandidateShift is a training objective for descriptor-free 2D–3D matching. The matcher still takes keypoint coordinates, landmark bearings, and RGB, and it is not changed at inference. During training, the same matcher sees a full candidate set and a reduced one. Geometric labels supervise whether any valid landmark survives. When one does, the full-set prediction is restricted to the retained landmarks, renormalized, and used as a stopped-gradient target for the reduced-set prediction. Both outputs also keep ordinary correspondence supervision. The method is tested on top of GeoMix and transferred to A2-GNN, mainly on MegaDepth, with zero-shot results on Cambridge and 7Scenes.

Critical strengths
  1. The training problem is stated precisely. Deleting landmarks can preserve a correspondence or remove every valid one, and the correct output in the second case is unmatched. CandidateShift uses geometric survival for that availability decision, and it transfers a soft identity target only for queries that remain matchable. That avoids teaching a preference among candidates that are all invalid. Inference adds no module.
Critical weaknesses
  1. Most of the reported gain comes from supervised candidate perturbations, and the two new terms do not play the roles claimed for them. Availability largely restates the unmatched label already present in the inherited correspondence loss, and the paper states that the ablation does not separate the two. The upper-quartile error does move further than the one-pixel AUC: top-1 translation at 75% falls from 2.15 m for GeoMix to 1.33 m with perturbations and about 0.89 m with identity or the full loss. That tail gain should be attributed to identity consistency, rather than to the joint availability-and-identity objective.
Questions for authors
  1. If the stopped-gradient target in Eq. 6 is replaced by the renormalized geometric correspondence labels, how do AUC and the deletion metrics in Figure 4 change?
  2. Are the Table 2 gaps stable across seeds? In particular, is the difference between identity-only and the full objective distinguishable from run-to-run variation?
  3. Which term accounts for the higher 7Scenes/SuperPoint error of CandidateShift (Mix), and does that ordering reverse for any loss variant?
8 comments
AuthorsOpenReview ↗

SjJD-W1 (1/2). Attribution of the pose-error tail improvement

We appreciate your careful analysis of the upper-quartile pose errors. It helped us explain more precisely how supervised perturbations and conditional identity transfer contribute to the improvement.

Reviewer concern.

Most of the reported gain comes from supervised candidate perturbations, and the two new terms do not play the roles claimed for them. Availability largely restates the unmatched label already present in the inherited correspondence loss, and the paper states that the ablation does not separate the two. The upper-quartile error does move further than the one-pixel AUC: top-1 translation at 75% falls from 2.15 m for GeoMix to 1.33 m with perturbations and about 0.89 m with identity or the full loss. That tail gain should be attributed to identity consistency, rather than to the joint availability-and-identity objective.

Response and evidence.

We agree that most of the further tail reduction beyond supervised perturbations is attributable to identity consistency. The revised Section 4.3 now states that identity provides the larger localization gain and most of the reduction in the error tail. The complete loss comparison is:

MegaDepth SIFT reprojection AUC (%).

ModelTop-1 AUC@1 / 5 / 10pxTop-10 AUC@1 / 5 / 10px
GeoMix (SIFT)14.99 / 47.35 / 54.1118.44 / 57.73 / 65.68
Perturbations only16.70 / 52.58 / 59.5220.61 / 62.66 / 70.51
Availability only17.43 / 54.31 / 61.2021.22 / 64.17 / 72.05
Identity only17.64 / 54.90 / 61.8621.49 / 64.80 / 72.75
CandidateShift17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
VariantTop-kRotation Quantile@25/50/75% (deg)Translation Quantile@25/50/75% (m)Pose failures
Perturbations onlyTop-10.0800 / 0.2600 / 12.58000.0076 / 0.0255 / 1.326045
Perturbations onlyTop-100.0500 / 0.1200 / 0.88000.0042 / 0.0113 / 0.08291
Availability onlyTop-10.0791 / 0.2307 / 10.32350.0073 / 0.0221 / 1.072739
Availability onlyTop-100.0485 / 0.1119 / 0.64820.0040 / 0.0107 / 0.06391
Identity onlyTop-10.0791 / 0.2256 / 8.50950.0072 / 0.0223 / 0.888644
Identity onlyTop-100.0485 / 0.1084 / 0.56990.0039 / 0.0103 / 0.05530
CandidateShiftTop-10.0791 / 0.2256 / 8.68450.0071 / 0.0220 / 0.886635
CandidateShiftTop-100.0485 / 0.1084 / 0.55990.0039 / 0.0101 / 0.05430
VariantTop-kPrecision (%)Recall (%)F1 (%)Correct matches/queryPose failures
Perturbations onlyTop-126.6231.7928.4547.7645
Availability onlyTop-127.9034.2730.2150.7139
Identity onlyTop-128.3734.6330.6451.0344
CandidateShiftTop-128.1634.5230.4750.8835
Perturbations onlyTop-1017.7746.4025.05146.641
Availability onlyTop-1018.4648.8426.13153.031
Identity onlyTop-1018.8749.1626.61153.660
CandidateShiftTop-1018.7549.1026.48153.430

The progression separates the benefit of candidate perturbations from the added pair supervision. Identity-only accounts for almost all of the further top-1 upper-quartile translation improvement; adding availability changes the translation tail only slightly while reducing invalid top-1 poses. The correspondence metrics also favor identity-only.

AuthorsOpenReview ↗

SjJD-W1 (2/2). Attribution of the pose-error tail improvement

The reweighting experiment now addresses the separate ambiguity concerning the event loss:

Revised Table 6 (p. 10).

Availability versus unmatched-loss reweighting. The availability column indicates the added event loss.

VariantAdded γAvailability lossTop-1 AUC@1/5/10px (%)Top-10 AUC@1/5/10px (%)
Perturbations only016.70 / 52.58 / 59.5220.61 / 62.66 / 70.51
Identity only017.64 / 54.90 / 61.8621.49 / 64.80 / 72.75
Perturbations+Identity0✓17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
Perturbations + reweighting416.42 / 52.07 / 58.8420.57 / 62.43 / 70.28
Identity + reweighting0.517.41 / 54.41 / 61.3521.48 / 64.63 / 72.55

Both selected reweighting controls have lower AUC than their corresponding unweighted objectives at all six entries. This clarifies the implemented loss comparison, while the multi-seed results in our response to SjJD-Q2 establish that the small identity-only/joint AUC differences should be interpreted separately from the larger gain over perturbations.

Changes in the revised manuscript. Section 4.3 (p. 7); Appendix G (p. 23); Table 18 (p. 20); Table 2 (p. 8); Table 6 (p. 10).

Table 2 compares the pair objective with supervised perturbations, holding the matcher, FiLM, and group weighting fixed. Both added terms improve localization; identity gives the larger gain and most of the error-tail reduction. The identity-only variant still uses the geometric survival gate: it removes Lavail\mathcal L_{\rm avail} while retaining the availability condition that defines valid identity targets.

Table 25 summarizes seeds 93, 94, and 95. Both identity-bearing objectives improve over supervised perturbations. Identity-only has higher mean AUC; CandidateShift has lower sample standard deviation than identity-only.

Table 26 compares models within each seed. The original seed-93 models remain the reference outside this section; the marked same-seed repeat is excluded from the summary statistics.

AuthorsOpenReview ↗

SjJD-Q1 (1/2). Geometric identity targets instead of a stopped soft teacher

We appreciate your targeted geometric-label alternative. It provides a direct test of whether the full-context soft prediction contributes information beyond another application of the correspondence labels.

Reviewer concern.

If the stopped-gradient target in Eq. 6 is replaced by the renormalized geometric correspondence labels, how do AUC and the deletion metrics in Figure 4 change?

Response and evidence.

We trained that alternative. For a query with at least one surviving geometric match, its identity target becomes

qiGT(j)=Yji∑ℓ∈JYℓi,j∈J.q_i^{\rm GT}(j)=\frac{Y_{ji}}{\sum_{\ell\in J}Y_{\ell i}},\qquad j\in J.

The survival condition, availability term, direct correspondence supervision, perturbations, FiLM, and group weighting remain the same. Normalizing the binary labels assigns equal mass to the geometrically valid retained landmarks when more than one exists.

MegaDepth SIFT reprojection AUC (%).

ModelTop-1 AUC@1 / 5 / 10pxTop-10 AUC@1 / 5 / 10px
CandidateShift17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
Geometric target17.42 / 54.02 / 60.7921.26 / 64.45 / 72.37

The stopped full-context target gives higher localization AUC at every threshold and both retrieval depths. Since both variants already receive geometric correspondence supervision, the result supports transferring relative preferences learned from the reference context rather than duplicating only the positive label set.

The complete candidate-deletion comparison is:

Candidate deletion on the fixed validation queries.

ModelRetained fractionConditional KL ↓Identity accuracy (%) ↑Availability accuracy (%) ↑Brier ↓BCE ↓Completed inputs
CandidateShift0.11.140040.9798.560.01510.0787503/512
CandidateShift0.20.945741.2197.150.02850.1269512/512
CandidateShift0.30.805841.8295.680.04050.1654512/512
CandidateShift0.40.662342.6494.320.05100.1969512/512
CandidateShift0.50.540943.4592.990.06030.2236512/512
CandidateShift0.60.425745.3991.790.06830.2460512/512
CandidateShift0.70.316946.5590.700.07550.2656512/512
CandidateShift0.80.217147.5089.630.08220.2836512/512
CandidateShift0.90.112848.3488.720.08820.2996512/512
Geometric target0.11.118142.6398.560.01520.0786503/512
Geometric target0.20.955540.8997.140.02850.1267512/512
Geometric target0.30.829641.5095.690.04050.1650512/512
Geometric target0.40.681241.4394.300.05100.1969512/512
Geometric target0.50.559042.8992.990.06040.2241512/512
Geometric target0.60.434544.7191.760.06850.2463512/512
Geometric target0.70.325445.7090.630.07580.2661512/512
Geometric target0.80.215847.0789.610.08240.2837512/512
Geometric target0.90.111147.8588.710.08820.2994512/512

At 10% retention, 503 of the 512 inputs complete the forward diagnostic; identity metrics further require surviving ground-truth matches. For CandidateShift, 499 images meet that identity-eligibility condition. At the other retained fractions all inputs complete. Identity accuracy tests the most probable real candidate only on queries with a valid surviving match. Availability metrics use all keypoints in the completed inputs, with ρi>0.5\rho_i>0.5 for the binary accuracy. Conditional KL is the full-to-reduced divergence between each model's restricted full prediction and its recomputed subset prediction, with unmatched entries removed from that conditional comparison.

CandidateShift has higher label-based identity accuracy from 20% through 90% retention; the geometric target is stronger at the most severe 10% retention. Conditional KL measures self-consistency rather than geometric correctness, so its ordering need not match identity accuracy at every fraction. Figure 4 now includes the geometric-target identity curves, and Figure 5 provides the complementary availability diagnostics.

Reviewer_rtgkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CandidateShift, a training strategy for descriptor-free 2D–3D matching under changes in candidate landmark sets. The authors observe that candidate deletion may remove all valid correspondences for a query keypoint, changing its correct prediction from a landmark match to an unmatched outcome. To address this issue, CandidateShift separates match availability from conditional landmark identity. It introduces a geometry-supervised availability loss and a consistency loss that transfers normalized matching distributions from the full candidate set to a reduced subset when valid correspondences remain. The method builds on existing matching architectures and incorporates supervised candidate perturbations, FiLM conditioning, and group DRO during training. Experiments on MegaDepth and cross-scene localization benchmarks show improvements in matching robustness and camera localization without adding inference overhead.

Critical strengths
  1. Candidate set variations are a practical concern in visual localization, where retrieval errors, map sampling, and outlier filtering can all change which landmarks are available to the matcher. The observation that removing candidates can change the correct output to "unmatched", rather than merely perturbing the input, provides a useful motivation for availability-aware supervision.

  2. The proposed losses are straightforward to integrate into existing matchers and do not introduce additional inference cost. Separating match availability from conditional identity also makes intuitive sense: there is little reason to enforce consistency among candidate identities once all valid correspondences have been removed. The reported transfer to A2-GNN further suggests that the training strategy is not restricted to a single matching backbone.

Critical weaknesses
  1. The methodological novelty is somewhat limited. At its core, CandidateShift combines explicit unmatched supervision with consistency regularization between predictions on full and reduced candidate sets. Both ideas have been explored in related matching and consistency-learning literature. Conditioning the identity loss on whether a valid match remains is sensible, but the overall formulation feels more like a task-specific adaptation of existing techniques than a substantial conceptual advance over perturbation-based training.

  2. The additional gains over the perturbation baseline are fairly small, and the ablation does not convincingly establish the value of the availability loss. In Table 2, supervised perturbations alone improve Top-1 AUC@1px from 14.99% to 16.70%, while the full CandidateShift objective reaches 17.69%. More importantly, the identity-only variant already achieves 17.64%, performing almost as well as the full method and outperforming it on several other metrics. This makes it difficult to determine how much the availability loss contributes beyond the conditional identity loss. The authors also acknowledge that their current ablations do not cleanly separate the effect of availability supervision from simply reweighting the matcher's existing unmatched (dustbin) loss.

Questions for authors
  1. What does the proposed availability loss capture beyond the unmatched supervision already provided by the underlying matcher? In particular, would adjusting the weight of the existing unmatched loss in the perturbation baseline produce a similar effect?

  2. In Table 2, the identity-only variant achieves 17.64%, nearly matching the full objective at 17.69% and performing better on some other metrics. Which results provide the strongest evidence that both losses are needed, rather than relying on conditional identity consistency alone?

8 comments
AuthorsOpenReview ↗

rtgk-W1 (1/2). Contribution relative to unmatched supervision and consistency learning

Thank you for asking us to locate the contribution more precisely relative to unmatched supervision and consistency learning. Your assessment helped us focus the revised account on the learning problem and the controls that test its construction.

Reviewer concern.

The methodological novelty is somewhat limited. At its core, CandidateShift combines explicit unmatched supervision with consistency regularization between predictions on full and reduced candidate sets. Both ideas have been explored in related matching and consistency-learning literature. Conditioning the identity loss on whether a valid match remains is sensible, but the overall formulation feels more like a task-specific adaptation of existing techniques than a substantial conceptual advance over perturbation-based training.

Response and evidence.

The contribution is a supervision relation between predictions whose candidate spaces have changed. An independently supervised matcher can learn that a reduced input is unmatched, but that supervision does not transfer the full-context preferences among identities that remain. Conversely, ordinary invariance does not specify what to do when deletion removes every valid identity. CandidateShift connects these cases through a known retained-index map and geometric survival.

For a full-set distribution pi\mathbf p_i, the deletion map preserves probabilities on retained identities and collects removed mass into the unmatched outcome. Its real-candidate conditional distribution provides the stopped identity target, while geometric labels determine whether a valid identity exists. The probabilistic separation can be expressed by the KL chain rule:

DKL(v∥w)=DKL(Bern⁡(ρv)∥Bern⁡(ρw))+ρvDKL(vˉ∥wˉ),D_{\rm KL}(\mathbf v\Vert\mathbf w)=D_{\rm KL}(\operatorname{Bern}(\rho_v)\Vert\operatorname{Bern}(\rho_w))+\rho_vD_{\rm KL}(\bar{\mathbf v}\Vert\bar{\mathbf w}),

where vˉ\bar{\mathbf v} and wˉ\bar{\mathbf w} condition on real candidates. This separates match existence from conditional identity. CandidateShift then uses geometric labels for the event and equal identity weight on surviving queries, instead of weighting identity by the teacher's retained mass as in the joint KL. This makes the survival decision depend on geometric correspondence rather than on a potentially mistaken teacher probability.

The new target and gating controls test the two design choices:

Revised Table 5 (p. 10).

Conditional identity target and survival gating.

VariantTop-1 AUC@1/5/10px (%) ↑Top-10 AUC@1/5/10px (%) ↑
CandidateShift17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
Geometric target17.42 / 54.02 / 60.7921.26 / 64.45 / 72.37
w/o survival gate17.76 / 54.78 / 61.5921.55 / 64.84 / 72.63

The restricted soft target improves localization over renormalized geometric labels at every threshold. Both models already receive direct geometric correspondence supervision, so the comparison tests the value of transferring the complete context's relative preferences. Removing the survival gate gives a smaller, mixed effect, with lower broad-threshold accuracy at both retrieval depths.

The availability/reweighting comparison addresses whether the result can instead be obtained by increasing the inherited unmatched penalty:

Revised Table 6 (p. 10).

Availability versus unmatched-loss reweighting. The availability column indicates the added event loss.

VariantAdded γAvailability lossTop-1 AUC@1/5/10px (%)Top-10 AUC@1/5/10px (%)
Perturbations only016.70 / 52.58 / 59.5220.61 / 62.66 / 70.51
Identity only017.64 / 54.90 / 61.8621.49 / 64.80 / 72.75
Perturbations+Identity0✓17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
Perturbations + reweighting416.42 / 52.07 / 58.8420.57 / 62.43 / 70.28
Identity + reweighting0.517.41 / 54.41 / 61.3521.48 / 64.63 / 72.55

Both selected reweighting controls have lower AUC than their corresponding unweighted objectives at all six entries. The pair objective also improves all six entries when FiLM and DRO are disabled:

MegaDepth SIFT reprojection AUC (%).

ModelTop-1 AUC@1 / 5 / 10pxTop-10 AUC@1 / 5 / 10px
w/o FiLM/DRO/pair16.77 / 53.59 / 60.5521.88 / 65.61 / 73.44
w/o FiLM/DRO17.62 / 54.91 / 61.9022.19 / 66.59 / 74.36
AuthorsOpenReview ↗

rtgk-W1 (2/2). Contribution relative to unmatched supervision and consistency learning

Finally, it transfers to a second matcher without changing its inference architecture:

MatcherTop-1 AUC@1/5/10px (%)Top-10 AUC@1/5/10px (%)
A2-GNN12.43 / 41.46 / 47.6116.59 / 52.86 / 60.82
A2-GNN + CandidateShift13.64 / 45.40 / 51.9419.19 / 59.46 / 67.08

CandidateShift transfers identity preferences across candidate sets whose geometric match availability changes. BCE and KL implement this supervision relation. Section 4.7 tests the target, survival condition, and event supervision against the alternatives suggested by the reviews.

Changes in the revised manuscript. Section 3 (p. 3); Section 3.3 (p. 4); Section 4.7 (p. 9); Appendix B.1 (p. 15); Appendix E.2 (p. 20); Appendix B (p. 15); Table 5 (p. 10); Table 6 (p. 10); Table 4 (p. 9).

Availability asks whether any geometrically valid landmark remains; identity asks which retained landmark matches the query. In Figure 2, the two branches use the same matcher. Retained indices align landmark identities, geometric labels determine survival, and only surviving queries receive the conditional identity target.

Since ρi=1−pi′(⊥)\rho_i=1-p_i^{\prime}(\bot), availability supervises the normalized dustbin event using labels from the existing geometric correspondence matrix. Compared with inherited transport supervision, it uses column normalization and class-balanced reduction on deletion pairs; Appendix B.1 tests explicit unmatched-loss reweighting.

Table 6 compares availability supervision with extra query-unmatched transport weight, selected independently by clean validation for each starting objective. Reweighting does not recover the results of the corresponding unweighted control or CandidateShift. Availability uses the same geometric matching event, but supervises its normalized probability with balanced available/unavailable classes. This separates the implemented event objective from increasing the inherited unmatched penalty.

AuthorsOpenReview ↗

rtgk-W2 (1/2). How much does availability contribute beyond identity?

We appreciate your careful comparison of the identity-only and joint objectives. It motivated us to examine the contribution of availability across localization accuracy, pose failures, and generalization.

Reviewer concern.

The additional gains over the perturbation baseline are fairly small, and the ablation does not convincingly establish the value of the availability loss. In Table 2, supervised perturbations alone improve Top-1 AUC@1px from 14.99% to 16.70%, while the full CandidateShift objective reaches 17.69%. More importantly, the identity-only variant already achieves 17.64%, performing almost as well as the full method and outperforming it on several other metrics. This makes it difficult to determine how much the availability loss contributes beyond the conditional identity loss. The authors also acknowledge that their current ablations do not cleanly separate the effect of availability supervision from simply reweighting the matcher's existing unmatched (dustbin) loss.

Response and evidence.

Conditional identity supplies the principal localization improvement beyond supervised perturbations. Availability alone also improves the perturbation reference, while their combination changes the balance between correspondence quality and pose outcomes:

MegaDepth SIFT reprojection AUC (%).

ModelTop-1 AUC@1 / 5 / 10pxTop-10 AUC@1 / 5 / 10px
Perturbations only16.70 / 52.58 / 59.5220.61 / 62.66 / 70.51
Availability only17.43 / 54.31 / 61.2021.22 / 64.17 / 72.05
Identity only17.64 / 54.90 / 61.8621.49 / 64.80 / 72.75
CandidateShift17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
VariantTop-kRotation Quantile@25/50/75% (deg)Translation Quantile@25/50/75% (m)Pose failures
Perturbations onlyTop-10.0800 / 0.2600 / 12.58000.0076 / 0.0255 / 1.326045
Perturbations onlyTop-100.0500 / 0.1200 / 0.88000.0042 / 0.0113 / 0.08291
Availability onlyTop-10.0791 / 0.2307 / 10.32350.0073 / 0.0221 / 1.072739
Availability onlyTop-100.0485 / 0.1119 / 0.64820.0040 / 0.0107 / 0.06391
Identity onlyTop-10.0791 / 0.2256 / 8.50950.0072 / 0.0223 / 0.888644
Identity onlyTop-100.0485 / 0.1084 / 0.56990.0039 / 0.0103 / 0.05530
CandidateShiftTop-10.0791 / 0.2256 / 8.68450.0071 / 0.0220 / 0.886635
CandidateShiftTop-100.0485 / 0.1084 / 0.55990.0039 / 0.0101 / 0.05430
VariantTop-kPrecision (%)Recall (%)F1 (%)Correct matches/queryPose failures
Perturbations onlyTop-126.6231.7928.4547.7645
Availability onlyTop-127.9034.2730.2150.7139
Identity onlyTop-128.3734.6330.6451.0344
CandidateShiftTop-128.1634.5230.4750.8835
Perturbations onlyTop-1017.7746.4025.05146.641
Availability onlyTop-1018.4648.8426.13153.031
Identity onlyTop-1018.8749.1626.61153.660
CandidateShiftTop-1018.7549.1026.48153.430

Identity-only accounts for most of the further reduction in the translation-error tail and has stronger matching precision/recall/F1. CandidateShift produces fewer invalid top-1 poses and somewhat lower translation quantiles. These measures capture different outcomes: more individually correct matches do not necessarily produce the best pose result for every query.

Reviewer_FzYyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses descriptor-free 2D-3D matching, namely, matching a query image to a sparse, colored 3D pointcloud with bearing vectors to a reference view. The authors argue that such a localizer has to distinguish whether a match remains available and to what 3D point it matches. By removing 3D point candidates, the authors note that the labels of match candidates also need to shift (from matched to unmatched), and that this changes the localization. They use this technique as an augmentation during training, forcing the network to make a decision on whether queries are matchable or not. Specifically, the authors propose an involved augmentation scheme including filtering 3D points, changing input context to a database image, photometric augmentations (setting a color channel to 0), FiLM conditioning and adaptive group supervision with DRO. They introduce additional loss terms on match availability (whether a query can be matched) and conditional identity. Evaluations on MegaDepth show that this improves localization accuracy over the baseline GeoMix by increasing the model's robustness. Additional benchmarks to unseen scenes (Cambridge, 7Scenes) show that the proposed training augmentations generalize, although not achieving SotA there.

Critical strengths
  • The paper addresses the interesting problem of direct 2D-3D matching. While most standard localization pipelines rely on image matching, this paper tackles the much harder problem of bypassing image matching and finding direct 2D-3D correspondences between an image and a colored point cloud.
  • The idea of using candidate deletion is technically sound. The paper describes that such 2D-3D matching models need to distinguish whether a 2D-3D correspondence is available. By removing 3D candidates during training, they force the network to make such decisions, increasing its robustness.
  • The evaluation is technically sound, and shows improvements over the GeoMix (SIFT) baseline. On MegaDepth, Cambridge, and 7Scenes, the proposed training scheme improves over the baseline method.
  • The ablation is clear and shows that perturbations, availability, and identity losses all improve robustness.
Critical weaknesses
  • The paper is hard to read, with many unnecessarily complicated terms. The pipeline figure is very hard to read because of many unclear mathematical terms only introduced later. The terms availability and identity are confusing, and only get clear much later in the paper. The presentation of different augmentations and losses is quite complicated, and some figures would help convey the message. Also, the many different sets introduced are confusing. Some figures would help here.
  • Despite many efforts, I have trouble understanding section 3.5. I still don't understand the role of FiLM adapation here. Why is it required? How does this help training? From the ablations (Fig. 4 and Table 2), I see that it helps, but I don't understand why. The section reads like what we do, but does not elaborate the reasoning behind it.
  • The approach is quite complicated for rather minor gains. While the evaluation shows gains over GeoMix (SIFT), the improvements are rather small, and the augmentation techniques are quite involved. They also hurt under low outlier contamination (Fig. 3). It also remains unclear whether these gains would not diminish under a larger training set (MegaDepth is quite small).
  • The many safeguards in mathematical formulas make it hard to read. I appreciate the authors' attention to detail, but the many safeguards (like the thresholds on the positive attention mass or the numerical floor in Eq. 1) make it hard to read. I would recommend moving these to the appendix to avoid hindering the reading flow of the main method section.
  • The improvements are rather small, and its not SotA on Cambridge and 7Scenes. Especially on 7Scenes, in the mixed setting, the proposed augmentations hurts accuracy.
  • The different perturbation groups are not ablated. The paper only ablates using all or none of the perturbation groups, so the impact of the independent perturbation groups remains unclear.
Questions for authors
  • Have you tried replacing the color channel augmentation on the query side with real photometric augmentations?
  • Could you elaborate why, in L_avail, you supervise the positive probability mass? What makes this better than just supervising the actual positive correspondence or the dustbin? By supervising the positive probability mass, also incorrect correspondences within the retained set get positive gradient, and thus noise.
  • Why does the proposed augmentation hurt reprojection accuracy at low outlier rates? Does the model learn a robustness-accuracy trade-off?
  • L164: "Outlier perturbation instead keeps all landmarks without a positive correspondence and subsamples those that have at least one." --> Isn't this inlier perturbation?
18 comments
AuthorsOpenReview ↗

FzYy-W1. Availability, identity, notation, and Figure 2

Thank you for identifying the points at which the exposition becomes difficult to follow. Your careful reading helped us introduce the two decisions before their notation and explain how to read the existing pipeline figure.

Reviewer concern.

The paper is hard to read, with many unnecessarily complicated terms. The pipeline figure is very hard to read because of many unclear mathematical terms only introduced later. The terms availability and identity are confusing, and only get clear much later in the paper. The presentation of different augmentations and losses is quite complicated, and some figures would help convey the message. Also, the many different sets introduced are confusing. Some figures would help here.

Response and evidence.

Availability asks whether any valid match remains; identity asks which retained landmark is the match. Consider a query with one correct landmark and two distractors. Deleting a distractor preserves the correct identity. Deleting its last valid landmark changes its correct output to unmatched. An identity distribution over the remaining landmarks still sums to one in both cases, so it cannot by itself express this difference.

We now introduce this distinction immediately before Figure 2. The figure keeps its original architecture and connections: the same matcher processes a reference set and its perturbed counterpart; retained indices align the output rows; recomputed geometric labels identify the queries for which conditional identity transfer is meaningful. The following reading guide summarizes the notation.

ObjectMeaningRole in the method
C,C′C,C'Reference and reduced candidate setsInputs to the shared matcher
JJRetained landmark indicesAligns identities across the two predictions
aia_iWhether query ii retains a geometrically valid matchAvailability target and identity eligibility
ρi\rho_iPredicted probability assigned to all real candidatesAvailability prediction
qi\mathbf q_iFull-context prediction restricted and renormalized on JJStopped conditional identity target
ri\mathbf r_iRecomputed subset prediction normalized on its real candidatesConditional identity prediction

The sets therefore implement a concrete correspondence relation between two outputs of different sizes. They distinguish queries that retain a valid match from those whose correct prediction changes. Section 3 introduces that relation, while Appendix A.1 collects the numerical implementation details. Figure 4 then shows how identity selection behaves when progressively fewer candidates are retained.

Changes in the revised manuscript. Section 3 (p. 3); Section 3.3 (p. 4); Section 3.4 (p. 5); Appendix B.2 (p. 16); Figure 2 (p. 3); Section 3.2 (p. 3); Appendix A.1 (p. 14); Figure 4 (p. 8).

Availability asks whether any geometrically valid landmark remains; identity asks which retained landmark matches the query. In Figure 2, the two branches use the same matcher. Retained indices align landmark identities, geometric labels determine survival, and only surviving queries receive the conditional identity target.

The geometric-target control uses qiGT(j)=Yji/∑ℓ∈JYℓiq_i^{\rm GT}(j)=Y_{ji}/\sum_{\ell\in J}Y_{\ell i} on surviving queries while retaining the other training terms. The ungated control instead retains the soft target but applies its identity loss even when ai=0a_i=0. Table 5 reports their localization results, and Figure 4 compares geometric correctness with self-consistency. When ai=0a_i=0, every retained landmark is invalid under the correspondence labels, so a normalized soft target still exists algebraically but does not describe a valid matching identity. The availability term remains defined for these queries.

AuthorsOpenReview ↗

FzYy-W2 (1/3). Why FiLM is used, and what the ablations actually show

We appreciate your close examination of Section 3.5 and the distinction between describing an operation and explaining why it is used. This prompted us to separate FiLM's motivation from the evidence for its effect.

Reviewer concern.

Despite many efforts, I have trouble understanding section 3.5. I still don't understand the role of FiLM adapation here. Why is it required? How does this help training? From the ablations (Fig. 4 and Table 2), I see that it helps, but I don't understand why. The section reads like what we do, but does not elaborate the reasoning behind it.

Response and evidence.

FiLM provides group-specific input conditioning for the perturbed branch. We use it to adapt coordinate and RGB statistics to the sampled corruption while sharing the reference encoder.

Retrieval, density, outlier composition, and color perturbations change different aspects of the input. The group-conditioned transformation is

hg(z)=z⊙[1+0.1tanh⁡(sg)]+0.1tanh⁡(bg).h_g(z)=z\odot[1+0.1\tanh(s_g)]+0.1\tanh(b_g).

It starts at the identity because the learned parameters are initialized to zero. Its scale stays in [0.9,1.1][0.9,1.1], and its shift stays in [−0.1,0.1][-0.1,0.1]. The input Jacobian is diagonal, with those bounded scales. It adjusts the coordinate/RGB inputs seen by the feature encoder while leaving the geometric labels and pose targets unchanged. The reference branch and deployed matcher bypass this operation.

The original loss ablations held FiLM fixed; they did not isolate its benefit. We have now completed that comparison:

Revised Table 7 (p. 14).

Auxiliary training components on MegaDepth.

VariantFiLMDROPairTop-1 AUC@1/5/10px (%) ↑Top-10 AUC@1/5/10px (%) ↑
CandidateShift✓✓✓17.69 / 54.78 / 61.6821.59 / 64.90 / 72.75
w/o FiLM✓✓17.69 / 55.63 / 62.6722.45 / 67.05 / 74.93
w/ uniform groups✓✓17.69 / 54.55 / 61.3521.67 / 65.08 / 72.94
w/o FiLM/DRO✓17.62 / 54.91 / 61.9022.19 / 66.59 / 74.36
w/o FiLM/DRO/pair16.77 / 53.59 / 60.5521.88 / 65.61 / 73.44

Removing FiLM improves source-domain localization. The target-domain tests show where conditioning helps instead:

Revised Table 14 (p. 18).

Cambridge localization after auxiliary-component removal. Cells: median translation/rotation (cm/°, ↓); Avg.: mean of unrounded scene medians.

VariantCambridge/SIFT King'sCambridge/SIFT HospitalCambridge/SIFT ShopCambridge/SIFT St Mary'sCambridge/SIFT Avg.Cambridge/SuperPoint King'sCambridge/SuperPoint HospitalCambridge/SuperPoint ShopCambridge/SuperPoint St Mary'sCambridge/SuperPoint Avg.
CandidateShift48/1.09118/3.6961/4.04412/11.47160/5.0715/0.3435/1.0410/0.8435/1.1024/0.83
w/o FiLM43/1.04101/2.8429/2.10298/8.93118/3.7314/0.3628/0.838/0.7728/0.9020/0.71
w/o FiLM/DRO43/0.98112/3.5041/3.31421/11.61154/4.8513/0.3535/0.857/0.8223/0.8020/0.71
AuthorsOpenReview ↗

FzYy-W2 (2/3). Why FiLM is used, and what the ablations actually show

Revised Table 15 (p. 18).

7Scenes localization after auxiliary-component removal. Cells: median translation/rotation (cm/°, ↓); Avg.: mean of unrounded scene medians.

Query detectorVariantChessFireHeadsOfficePumpkinKitchenStairsAvg.
SIFTCandidateShift3/1.334/1.553/2.295/1.467/1.777/1.8739/9.4410/2.82
SIFTw/o FiLM3/1.354/1.593/2.285/1.476/1.767/1.8741/9.2610/2.80
SIFTw/o FiLM/DRO3/1.354/1.583/2.325/1.477/1.777/1.9146/10.3311/2.96
SuperPointCandidateShift3/1.375/1.672/2.165/1.418/2.047/1.9139/7.9110/2.64
SuperPointw/o FiLM3/1.375/1.752/2.235/1.428/2.007/1.9340/8.3910/2.73
SuperPointw/o FiLM/DRO3/1.365/1.743/2.225/1.408/1.997/1.9242/8.6710/2.76

Removing FiLM improves MegaDepth and Cambridge but worsens the 7Scenes/SuperPoint scene averages. Removing DRO as well retains this indoor disadvantage. With both auxiliary components disabled, the paired comparison in Table 7 shows that adding the pair objective improves all six AUC entries.

Recall complements these median errors by retaining unsuccessful queries in its denominator:

Revised Table 16 (p. 19).

Localization recall (%) and failures after auxiliary-component removal.

DatasetVariantSIFT 5cm/5°SIFT 25cm/2°SIFT 50cm/5°SIFT 1m/10°SIFT FailuresSuperPoint 5cm/5°SuperPoint 25cm/2°SuperPoint 50cm/5°SuperPoint 1m/10°SuperPoint Failures
CambridgeCandidateShift0.9514.4231.2650.78011.3154.2369.9577.810
Cambridgew/o FiLM0.8617.7036.4453.63013.8256.5671.8580.480
Cambridgew/o FiLM/DRO1.3017.5336.0151.21013.1357.7772.2881.950
7ScenesCandidateShift47.1059.7682.6286.38046.7659.9884.9489.380
7Scenesw/o FiLM46.9959.4982.8586.73045.4658.8884.1489.110
7Scenesw/o FiLM/DRO46.8159.0782.2485.79046.5859.3483.8588.310

On 7Scenes/SuperPoint, FiLM removal reduces recall at all four pose thresholds as well as worsening the mean scene-median errors. We retain FiLM in the main configuration for this indoor SuperPoint transfer benefit.

We also audited the group-weight dynamics. The multiplicative update accumulates detached supervised-loss differences, with no forgetting term, and the outlier-composition group dominates the weights. At the observed finite-precision limit, the denominator floor can nearly suppress shifted supervision when that group is absent; the clean and pair losses remain outside this weighting. Appendix A.2 reports this behavior. The uniform-weight control and the paired comparison without both auxiliaries assess the effects of these training choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.