acceptodds
Under review as a conference paper at ICLR 2027

UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching

Abstract

Despite recent advances in accuracy and efficiency, coarse matching remains an indispensable yet costly stage in existing semi-dense matchers due to dense token-level matching. We present UltraMatch, an ultra-efficient and scalable semi-dense matching framework that bypasses the quadratic computation and memory cost of dense token-level matching by routing only a small fraction of candidate matching paths. At its core, a lightweight Transport Path Router operates on coarse block representations to rank candidate target blocks for each source block and retain only a small set, restricting subsequent token-level matching to the selected paths and avoiding the construction of the full token-to-token matching matrix. We further design a sparse global Dual-Softmax that performs matching only over the routed block candidates while retaining global competition across the sparse matching space. Beyond matching acceleration, UltraMatch employs deployment-oriented structural reparameterization for feature extraction and a tiny fine matching head with shared parameters, further reducing inference cost and memory consumption. UltraMatch achieves competitive accuracy among semi-dense matchers, while running 1.67 faster than SuperPoint+LightGlue with only 0.44 GiB peak inference memory. Its scalability enables inference at up to 6K resolution on a single RTX 3090, whereas existing semi-dense matchers run out of memory before reaching 2K. Our routing strategy is also transferable, delivering about 2 end-to-end speedup in EDM and ELoFTR without accuracy loss.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semi-dense matching, feature matching, correspondence learning, Efficient Image Matching

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PgGsOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents UltraMatch, an efficient semi-dense image matching framework. A block-level router uses low-resolution interacted features to select a small set of candidate target regions before token-level coarse matching. Dual-Softmax is then applied over the retained sparse correspondence graph, preserving competition across routed regions. Combined with a reparameterized backbone and a lightweight refinement head, the framework achieves competitive geometric accuracy with substantial runtime and memory savings. The paper also evaluates the routing strategy within other matching architectures and demonstrates inference at high input resolutions.

Critical strengths
  1. Preserving cross-region matching competition is a meaningful design choice. Source tokens from different blocks may compete for the same target token, so independently normalizing each routed region can lose important matching constraints. The proposed sparse global normalization retains these interactions among the selected candidates. The ablation results support this choice, with AUC@5 decreasing from 57.4 to 54.1 when normalization is performed independently within routed blocks.
  2. The efficiency improvements are substantial and supported by several experiments. Within the same architecture, replacing routing with dense matching increases runtime from 32.26 ms to 73.78 ms. Transfer experiments with retrained variants of existing matchers provide further evidence that the benefit extends beyond the proposed backbone. The full-FP32 comparison also supports gains beyond mixed-precision optimization, while the high-resolution experiments demonstrate practical scalability.
Critical weaknesses
  1. The discussion and comparison with existing coarse-matching sparsification approaches are insufficient. Comparisons against dense matching establish that candidate reduction is effective, but do not sufficiently establish the advantage over existing feature filtering or straightforward hierarchical candidate selection. A comparison under comparable training and computational budgets would help isolate the contribution of the proposed routing formulation.
  2. The conceptual novelty appears limited. The routing stage can be understood as an additional, coarser matching stage preceding token-level matching and refinement. Although the accompanying supervision, halo expansion, and sparse normalization are empirically useful, the paper could more clearly explain what new algorithmic insight the combination provides beyond a conventional hierarchical matching strategy.
  3. The presentation is occasionally indirect. Some descriptions rely on abstract terminology where concrete operations and tensor dimensions would be more informative.
Questions for authors

Clarification requests:

  1. What exactly does �0��0� implement in Eq. (1)? Please specify the operation and input/output spatial dimensions. With the default �1��1�, the routing grid appears to have the same spatial resolution as the �2��2� features. Is �3��3� an identity operation in this setting, or does it perform additional resampling or alignment?

  2. How sensitive is the reported speed advantage to the streaming dense baseline’s implementation? The baseline uses a two-pass, chunked PyTorch implementation. While it provides a useful memory-efficient reference, its latency does not establish that efficient dense implementations necessarily incur comparable overhead. Please clarify the implementation bottlenecks and distinguish the benefit of candidate sparsification from implementation overhead.

  3. Could the authors explicitly qualify the approximately 2K OOM claim as applying to the tested official configurations? Table 9 shows that streaming dense matching reduces memory to 0.43 GiB, comparable to routing’s 0.44 GiB at the evaluated resolution. The observed OOM thresholds should therefore be distinguished from limitations intrinsic to the matching algorithms.

Words to authors: I lean toward acceptance given the substantial efficiency gains and supporting ablations, although the novelty and the attribution of the memory savings remain insufficiently clear. Satisfactory answers to the questions above could lead me to raise my recommendation to clear accept.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PgGs (1/2): Sparsification baselines and routing contributions

Thank you for recognizing our scalability and efficiency evaluation. We clarify the comparison with simpler sparsification strategies, the contribution beyond hierarchical search, the concrete routing operations, and the interpretation of the streaming and memory results below. We address your comments in two parts.

W1. Comparison with Simpler Sparsification Strategies

Same-Architecture Comparisons. We compare UltraMatch with pooled-feature block similarity followed by Top-RR selection, SLiM-style feature-norm filtering, and dense matching within the UltraMatch architecture on MegaDepth. The pooled baseline uses the same routing size R=6R=6; norm filtering retains tokens on both sides before matching the retained tokens.

Matching strategyAUC@5 / @10 (%) ↑Token-similarity GFLOPs ↓Time (ms/pair) ↓Peak memory (GiB) ↓
UltraMatch, R=6R=657.4 / 72.52.2931.190.43
Pooled-similarity Top-RR, R=6R=655.2 / 71.42.2931.490.43
Norm filtering, α=0.5\alpha=0.555.5 / 71.739.4645.703.34
Norm filtering, α=0.65\alpha=0.6555.6 / 72.224.9940.242.70
Norm filtering, α=0.8\alpha=0.853.7 / 70.516.7637.771.92
Dense matching56.5 / 72.4220.1583.398.26

Summary. Pooled Top-RR provides a useful control with the same candidate set size and nearly identical runtime and memory. Norm filtering reduces dense matching cost, but still requires 16.76–39.46 GFLOPs and 1.92–3.34 GiB, compared with 2.29 GFLOPs and 0.43 GiB for routing. Relative to dense execution, routing reduces token-similarity computation by 96.1×. The retrained ablations below further isolate the contributions of halo expansion and sparse global normalization.


W2. Contribution beyond Hierarchical Search

Our strategy builds on the hierarchical principle of multiscale optimization. We translate this principle into a lightweight Transport Path Router, Sparse Global Dual-Softmax, and halo expansion to overcome the computation and memory bottlenecks of dense coarse matching in semi-dense pipelines.

Routing and Boundary Coverage. The router uses the interacted 1/321/32 features to select a small set of target blocks for 1/81/8 token matching. Similarities are evaluated only along these selected paths, avoiding construction of the full token-to-token matrix. A one-token halo extends each selected block to retain valid correspondences near its boundaries with limited additional computation.

Sparse Global Dual-Softmax. Our normalization preserves competition across source blocks while operating only on the routed token pairs, maintaining strong matching performance at low computational cost. Table 4 shows that global normalization substantially improves accuracy over local normalization at essentially unchanged latency. The halo ablation further confirms the benefit of preserving boundary correspondences.

Retrained variant (Table 4)AUC@5 (%) ↑AUC@10 (%) ↑Time (ms) ↓
Dense 1/8 matching55.871.973.78
Local Dual-Softmax instead of Sparse Global DS54.169.732.68
Without halo54.671.031.77
Full UltraMatch57.472.532.26
AuthorsOpenReview ↗

Response to Reviewer PgGs (2/2): Implementation details, streaming and memory limits

W3 / Q1. Concrete Operations and Tensor Dimensions

Thank you for pointing out this ambiguity and the need for more concrete descriptions.

Eq. (1). With the default B=4B=4, the routing grid has spatial dimensions (H/8)/B×(W/8)/B=H/32×W/32(H/8)/B\times(W/8)/B=H/32\times W/32, exactly matching the interacted features F~32\widetilde{F}^{32}. Therefore, A=Id\mathcal{A}=\mathrm{Id}: it directly passes these features through, without pooling, interpolation, resampling, or additional alignment. We retain A\mathcal{A} in Eq. (1) solely for generality of the formulation. Both its input and output have shape [n,256,H/32,W/32][n,256,H/32,W/32], where nn is the batch size and H×WH\times W is the preprocessed image size.

The subsequent ϕ\phi is a shared 1×11\times1 Conv–BN projection from 256 to 64 channels, producing [n,64,H/32,W/32][n,64,H/32,W/32] without changing spatial resolution. Flattening the spatial dimensions yields [n,M,64][n,M,64], where M=(H/32)(W/32)M=(H/32)(W/32); Norm2\mathrm{Norm}_2 then applies ℓ2\ell_2 normalization along each descriptor's 64 channels.

We have clarified the identity alignment in Sec. 3.6 and the flattened tensor shape and normalization dimension in Appendix A of the revised manuscript.


Q2. Streaming Implementation Sensitivity and Efficiency Attribution

The 289.22 ms streaming result at 1152×1152 in Table 9 of the original submission (Table 11 in the revision) is specific to our two-pass PyTorch implementation. We additionally vary tile size, retaining complete global Dual-Softmax and two passes over all token similarities. Inputs are resized and square-padded to the listed dimensions.

Implementation1152×1152: time (ms) ↓1152×1152: memory (GiB) ↓2048×2048: time (ms) ↓2048×2048: memory (GiB) ↓
Native dense74.168.26OOMOOM
Streaming dense, tile 512990.710.439396.980.94
Streaming dense, tile 1024285.090.432415.140.94
Streaming dense, tile 2048148.960.431293.670.94
Streaming dense, tile 4096136.240.571168.800.94
UltraMatch, R=6R=631.190.4374.530.97

Implementation Overhead. Increasing the tile size reduces streaming latency from 990.71 to 136.24 ms at 1152×1152, confirming substantial implementation sensitivity. The two-pass implementation recomputes dense similarities, while chunking introduces per-tile reductions and kernel launches. Against the fastest tested streaming setting, routing is 4.37× faster at 1152×1152 and 15.68× faster at 2048×2048.

Candidate Reduction. Both native and streaming dense matching evaluate the complete token-pair space. Routing directly reduces that computation, as quantified by the similarity GFLOPs in our W1 response (Part 1). Independently of streaming, the retrained native-dense comparison in Table 4 gives 73.78 → 32.26 ms (2.29×) at 1152×1152.


Q3. Scope of the OOM Claim

We have qualified the OOM statements in the revised abstract and introduction to refer to the official implementations.

This is an observed implementation limit. Streaming dense matching successfully runs at 2048×2048 with 0.94 GiB, slightly below routing's 0.97 GiB, demonstrating that avoiding full-matrix storage can remove this memory bottleneck. Routing additionally reduces the number of evaluated candidates, achieving 74.53 ms versus 1168.80 ms for the fastest tested streaming configuration. The memory bottlenecks in the tested official implementations can indeed be addressed using additional techniques such as streaming matching. However, our router achieves ultra-low memory consumption and very fast inference simultaneously by reducing candidate computation while avoiding full-matrix storage.

Thank you again for your valuable feedback. We welcome any further questions and discussion.

Thank you for your positive reassessment of our work and for your time and consideration.

Reviewer_kPQ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an efficient semi-dense matching approach for high resolution images. The idea is centered on Transport Path Routing that finds candidate matches on coarse level and follows with token-level sparse Dual-Softmax. Structural reparametrization is applied for efficient feature extraction. An additional head is used for subpixel refinement. The evaluation shows significant improvement in terms of training and inference time and memory which allows processing of high resolution images (up to 6k) that is 3 times higher than competing methods with similar compute budget.

Critical strengths

The proposed Transport Path Routing, inspired by optimal transport principle, is applicable to various semi-dense matching architectures and leads to significant reduction in training cost, memory requirements and inference time. The approach is well motivated theoretically and empirically. The routing combined with other methods ELoFTR, JamMA, EDM leads to 2x acceleration without sacrificing their matching accuracy which is measured by downstream tasks such as pose estimation and visual localization and homography estimation on standard benchmarks. The datasets include ScanNet, MegaDepth, HPatches, Aachen, InLoc. The significant improvement in efficiency while keeping the accuracy allows to expand the applicability of semi-dense matching to high resolution of up to 6k on lower end hardware GPUs. The method is well presented with essential formulations understandable for readers familiar with efficient image matching techniques. The ablation study on pose estimation provides details of loss in accuracy when removing different components of the approach as well as the associated time budget. Overall the approach is convincing and more importantly can be adopted by other methods.

Critical weaknesses
  1. The evaluation could have included more comparison to controlled baselines for efficient sparse matching, eg simple candidate block selection by ranking pooled-feature blocks, feature filtering like in SLiM, or coarse to fine search.

  2. The scalability and efficiency is well evaluated but the loss in matching accuracy could have been investigated further. Using pose as a proxy metric obscures the drop in density and coverage of matches, accurate pose can be recovered from few matches. It would be interesting to see the breaking point with increasing pruning/decreasing routing size.

  3. Multiview reconstruction accuracy would be a very valuable addition to the experimental evaluation, it would test whether pruning preserves enough distributed correspondences to build a complete model, beyond recovering a correct pairwise pose, eg high resolution reconstruction on ETH3D to see if the efficiency gains survive the integration into a full reconstruction pipeline. A larger evaluation on difficult low-overlap pairs would make it more convincing eg. on WxBS to measure correspondence accuracy, coverage across routing size.

Questions for authors

Rather than rephrasing the critical points into questions which are mostly about the evaluation lease address weaknesses 1-3. Are there any available results that address these concerns? It would also be useful to have some visual examples of what type of correspondences the routing discards?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kPQ1 (1/2): Sparsification baselines and match quality

Thank you for recognizing our scalability and efficiency evaluation. We agree that pose accuracy alone cannot establish whether pruning preserves sufficiently dense and distributed correspondences. We provide controlled sparsification comparisons, direct matching statistics across routing sizes, reconstruction results, and an analysis of discarded correspondences below. We address your comments in two parts.

W1. Controlled Baselines for Efficient Sparse Matching

Comparison of Sparse Matching Strategies. We compare UltraMatch with pooled-feature block similarity followed by Top-RR selection, SLiM-style feature-norm filtering, and dense matching within the UltraMatch architecture. Pooled Top-RR is a simple hierarchical candidate-selection baseline with the same R=6R=6 size.

Matching strategyAUC@5 / @10 (%) ↑Token-similarity GFLOPs ↓Time (ms/pair) ↓Peak memory (GiB) ↓
UltraMatch, R=6R=657.4 / 72.52.2931.190.43
Pooled-similarity Top-RR, R=6R=655.2 / 71.42.2931.490.43
Norm filtering, α=0.5\alpha=0.555.5 / 71.739.4645.703.34
Norm filtering, α=0.65\alpha=0.6555.6 / 72.224.9940.242.70
Norm filtering, α=0.8\alpha=0.853.7 / 70.516.7637.771.92
Dense matching56.5 / 72.4220.1583.398.26

Benefit beyond Simple Candidate Selection. UltraMatch's reported AUC@5 / @10 exceeds pooled Top-RR by 2.2 / 1.1 percentage points at the same routing size and token-similarity cost. Compared with dense matching, it achieves a 2.67× speedup, with 99.0% less token-similarity computation and 94.8% lower peak memory. It also achieves higher AUC than the tested norm-filtering settings with lower computational and memory costs.


W2. Match Density, Coverage, and the Effect of Pruning

Evaluation beyond Pose AUC. We evaluate R∈{1,2,4,6,8,12,16}R\in\{1,2,4,6,8,12,16\} and dense inference on MegaDepth-1500. We report final match counts, correct-match counts, matching precision, final spatial coverage, and GT coverage separately. GT coverage measures whether a valid coarse correspondence remains available to the matcher; it does not measure the spatial distribution of the final output. Counts are averages per pair.

Routing size / modeAUC@5 / @10 (%) ↑Final / correct matches ↑Precision (%) ↑Final spatial coverage (%) ↑GT coverage (%) ↑Time (ms) ↓Memory (GiB) ↓
R=1R=154.5 / 70.93714 / 358194.1646.6782.230.410.43
R=2R=254.7 / 71.03948 / 384095.2746.84—30.430.43
R=4R=455.6 / 71.73988 / 390496.3046.5297.930.370.43
R=6R=657.4 / 72.53973 / 389996.7346.2798.831.190.43
R=8R=856.6 / 71.93960 / 389196.9646.1299.231.820.43
R=12R=1256.2 / 72.33944 / 388097.2245.92—33.930.52
R=16R=1656.7 / 72.63933 / 387297.3545.80—36.080.61
Dense56.5 / 72.43827 / 378097.8744.77100.083.398.26

Where Aggressive Pruning Becomes Costly. At R=6R=6, the final output has 3899 correct matches per pair and 46.27% spatial coverage, compared with 3780 and 44.77% for dense inference, although precision is lower (96.73% vs. 97.87%). Thus, the efficiency gain does not entail a reduction in average final match count or spatial coverage in this comparison.

Reducing the routing size to R=1R=1 lowers GT coverage from 98.8% to 82.2%, correct-match count from 3899 to 3581, and precision from 96.73% to 94.16%. This costs 8.2% of correct matches for only a 2.5% latency reduction, with no measured peak-memory saving. Spatial coverage remains high.

Why More Candidates Do Not Guarantee Higher Pose Accuracy. Increasing RR improves GT coverage but also broadens competition in Sparse Global Dual-Softmax. Standard expansion from R=6R=6 to R=8R=8 reduces incorrect matches from approximately 74 to 69 per pair, calculated from the table's rounded counts, while correct matches decrease slightly from 3899 to 3891 and pose AUC declines.

With features, raw similarities, and eligible output candidates fixed at R=6R=6, we vary only the Dual-Softmax normalization range.

Normalization rangeMean DS score of the same GT-correct correspondencesCorrect matches / pairCorrect-match spatial coverage (%)
R=6R=60.36873898.646.27
R=8R=80.36693877.345.97

Broader normalization weakens correct correspondences and can contribute to the AUC decline.

AuthorsOpenReview ↗

Response to Reviewer kPQ1 (2/2): Reconstruction, challenging pairs and discarded matches

W3. Reconstruction and Challenging Image Pairs

ETH3D Reconstruction Quality and Efficiency. We evaluate UltraMatch with R=1,6,8R=1,6,8 and EDM in the reconstruction pipeline. Quality metrics are percentages; times are in minutes per scene. Memory denotes peak process memory for reconstruction, separate from the pairwise matcher memory in Part 1.

MethodF1@1 cm ↑F1@2 cm ↑F1@5 cm ↑Accuracy@2 cm ↑Completeness@2 cm ↑Matching time (min) ↓Full-pipeline time (min) ↓Peak process memory (GiB) ↓
UltraMatch, R=6R=632.1451.3269.6687.1937.8111.5229.431.58
UltraMatch, R=1R=131.2949.0167.3381.6536.0611.4828.581.62
UltraMatch, R=8R=832.0651.6469.7486.4438.2111.3929.221.63
EDM32.4250.0766.7582.0937.4513.5631.5821.54

At R=6R=6, UltraMatch improves F1@2 cm and F1@5 cm over EDM by 1.25 and 2.91 percentage points. These results directly test the effect of pruning on the recovered model. Relative to EDM, R=6R=6 reduces matching time by 15.0%, full-pipeline time by 6.8%, and peak process memory by 92.7%. The efficiency benefit therefore remains measurable after integration.

Low-Overlap Evaluation. We evaluate the MegaDepth-1500 subset with GT overlap scores of 0.1–0.3, comprising 600 pairs from two scenes, 300 per scene. The table below separates final correct-match spatial coverage from GT coverage, reported as both the mean per-pair retention rate and the pooled retention rate across all GT correspondences.

Routing size / modeAUC@5 / @10 (%) ↑Correct matches ↑Precision (%) ↑Correct-match spatial coverage (%) ↑GT coverage, per-pair / pooled (%) ↑
Dense63.2 / 77.9295096.5140.08100.00 / 100.00
R=1R=160.0 / 75.4282590.8542.1673.45 / 79.46
R=2R=261.6 / 76.6302792.3242.4785.07 / 90.89
R=4R=460.5 / 76.0307993.8942.1592.61 / 96.84
R=6R=662.2 / 77.1307294.6041.8995.10 / 98.18
R=8R=861.1 / 76.8306494.9941.7296.38 / 98.75
R=12R=1261.7 / 77.4305295.4541.4797.66 / 99.24
R=16R=1662.5 / 76.8304495.6641.3198.31 / 99.44

On these low-overlap pairs, R=6R=6 retains 98.18% of pooled GT correspondences and provides more correct final matches and greater correct-match spatial coverage than dense inference.

WxBS. We additionally evaluate the official WxBS v1.1 test set, using a longest image side of 1152 pixels. The metric is the recall of manually annotated GT correspondences under the estimated fundamental matrix at the specified pixel threshold, averaged over image pairs. We compare EDM and SLiM with UltraMatch across routing sizes R∈{2,4,6,8}R\in\{2,4,6,8\}.

MetricUltraMatch, R=2R=2UltraMatch, R=4R=4UltraMatch, R=6R=6UltraMatch, R=8R=8EDMSLiM
Recall@1 px (%) ↑22.2425.8125.6326.5325.3619.37
Recall@3 px (%) ↑47.4550.2550.9752.3052.0740.32

Within the routing-size sweep, R=8R=8 achieves the highest recall at both thresholds. These results assess geometric estimation across routing sizes under challenging image changes, complementing the direct match-density and spatial-coverage analysis in Part 1.


Q1. What Correspondences Does Routing Discard?

We provide 12 groups of visualizations in the supplementary material. Each group contains four panels: (a) final UltraMatch matches at R=6R=6, colored by confidence; (b) epipolar-correct dense final matches absent from the routed output (green); (c) epipolar-incorrect dense final matches absent from the routed output (red); and (d) GT coarse correspondences excluded from the R=6R=6 routing candidate set (orange). These examples illustrate which correspondences are excluded in individual image pairs. However, we cannot currently conclude from these results that the router systematically discards a particular type of correspondence.

Thank you again for your valuable feedback. We welcome any further questions and discussion.

Thank you, I now have enough information to make the recommendation.

Reviewer_NQGUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the computational and memory costs of dense coarse matching in semi-dense image matching. It uses low-resolution, cross-image features to route each source block to a small set of target blocks, then performs token matching and global dual-softmax normalization within the resulting sparse candidate graph. Structural reparameterization and a lightweight refinement head with shared parameters further improve efficiency. The method aims to retain competitive geometric accuracy while supporting higher-resolution inputs.

Critical strengths
  • The accuracy–efficiency trade-off is attractive. On MegaDepth, UltraMatch achieves an AUC@5° of 57.4, close to EDM’s 57.5, while running about 2.6× faster and using only 0.44 GiB of peak memory under the reported settings. The advantage remains under full FP32 inference, suggesting that the gains extend beyond the choice of numerical precision.
  • The routing component has supporting evidence beyond full-model comparisons. Replacing it with dense matching within the same architecture more than doubles runtime, while integrating it into ELoFTR and EDM yields approximately twofold overall speedups. The paper also ablates global normalization, halo expansion, and routing supervision.
  • The evaluation covers relative pose estimation, homography estimation, indoor and outdoor visual localization, and high-resolution scaling. Processing 6K image pairs on a single RTX 3090 demonstrates useful scalability.
Critical weaknesses
  • I would appreciate a clearer discussion of the added value over existing filtering strategies. My understanding is that the feature interaction and refinement designs build substantially on EDM, with the main addition being routing at the coarse matching stage. SLiM reports a 1.83× speedup for ELoFTR through norm-based filtering without retraining, while Table 3 here reports a 1.85× speedup with retraining. These numbers come from different experimental settings, so they do not establish a direct ranking. Still, they raise an interesting ablation about where learned routing offers additional benefits over this simpler alternative, particularly in memory usage, accuracy preservation, and high-resolution scaling. A comparison under the same ELoFTR configuration would help clarify these trade-offs and the added contribution.

  • Low-overlap pairs tend to offer fewer valid correspondences, making the retention of useful matches particularly important for pose estimation. I'm wondering whether a fixed top-R routing budget remains reliable on these harder pairs, or whether it sometimes removes correspondences that matter for estimating the pose. The 98.8% aggregate ground-truth coverage in Table 5 is encouraging, although I could not find results grouped by image overlap.

  • The retrained baselines in Table 3 are somewhat weaker than those in Table 1; for example, EDM's AUC@5° changes from 57.5 to 55.4. I appreciate that Appendix B.3 describes the retraining protocol and that the baseline and routed variants use matched training settings. Could the authors clarify which checkpoints and training settings are used for the main comparison, and what accounts for this accuracy gap? This would help put the transfer results in context.

Questions for authors
  • One result I found interesting is that increasing the routing budget beyond the default in Table 5 improves ground-truth coverage, while AUC@5° falls from 57.4 to 56.6. Do the authors have an explanation for this trend? For example, are there corresponding changes in inlier ratios, incorrect matches, or normalized matching scores? Understanding this would also help explain why candidate pruning can preserve or even improve accuracy.

  • Could the authors compare the routed and dense variants of the same architecture on low-overlap subsets, such as the 0.1–0.3 range, reporting pose AUC and the fraction of ground-truth correspondences retained? It would also help to specify the overlap definition and number of pairs in each group. This would clarify whether the strong overall results also hold in this more challenging setting.

  • The localization experiments are useful, and I was curious about the InLoc DUC2 results, where UltraMatch performs below ELoFTR across the reported thresholds. Have the authors noticed any characteristic failure cases or differences in the spatial distribution of matches? A brief discussion would help clarify where the method works well and where difficulties remain.

  • Do the authors plan to release the code and pretrained models? Given the low memory footprint and support for high-resolution inputs, I could see an open implementation being useful for high-resolution matching and localization on edge devices or other hardware with limited compute and memory.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NQGU (1/3): Sparsification comparisons and resolution scaling

Thank you for recognizing our scalability and efficiency evaluation. We provide controlled filtering comparisons, results grouped by image overlap, clarification of the retraining protocol, an analysis of the routing-size trade-off, and localization diagnostics below. We address your comments in three parts.

W1. Added Value over Norm-Based Filtering

To provide a fairer comparison of routing and simpler filtering strategies, we add controlled experiments within the same host architectures. These experiments compare accuracy, runtime, memory usage, and high-resolution scaling while retaining each host architecture's feature extraction, interaction, and refinement modules.

UltraMatch Comparisons. On MegaDepth-1500, we compare routing with pooled-feature block similarity followed by Top-RR selection, SLiM-style feature-norm filtering, and dense inference within the UltraMatch architecture. Images are resized and square-padded to 1152×1152.

Matching strategyAUC@5 / @10 (%) ↑Token-similarity GFLOPs ↓Time (ms/pair) ↓Peak memory (GiB) ↓
UltraMatch, R=6R=657.4 / 72.52.2931.190.43
Pooled-similarity Top-RR, R=6R=655.2 / 71.42.2931.490.43
Norm filtering, α=0.5\alpha=0.555.5 / 71.739.4645.703.34
Norm filtering, α=0.65\alpha=0.6555.6 / 72.224.9940.242.70
Norm filtering, α=0.8\alpha=0.853.7 / 70.516.7637.771.92
Dense matching56.5 / 72.4220.1583.398.26

Routing improves AUC@5 / @10 over pooled Top-RR by 2.2 / 1.1 percentage points at the same routing size and token-similarity cost. Compared with dense inference, it reduces token-similarity computation by 96.1×, runtime by 2.67×, and peak memory by 94.8%, while preserving pose accuracy in this comparison. Norm filtering also reduces computation, but the tested settings require 16.76–39.46 GFLOPs and 1.92–3.34 GiB, compared with 2.29 GFLOPs and 0.43 GiB for routing.

ELoFTR Norm-Filtering Control. We additionally evaluate dense matching and norm filtering within ELoFTR under the same FP16 AMP configuration.

ELoFTR matching strategyAUC@5 / @10 (%) ↑Time (ms/pair) ↓Coarse matching (ms/pair) ↓Peak memory (GiB) ↓
ELoFTR54.9 / 71.2141.6177.888.64
+ Routing55.1 / 71.976.417.462.02
Dense matching (AMP)54.6 / 71.1143.2777.836.96
Norm filtering, α=0.5\alpha=0.5 (AMP)49.1 / 66.489.0925.324.73
Norm filtering, α=0.65\alpha=0.65 (AMP)48.8 / 66.580.7416.954.33
Norm filtering, α=0.8\alpha=0.8 (AMP)48.9 / 66.477.1913.223.39

Under AMP, norm filtering reduces runtime and memory but lowers AUC@5 by 5.5–5.8 percentage points. Routing improves AUC@5 / @10 by 0.2 / 0.7 percentage points, provides 1.85× end-to-end and 10.44× coarse-matching speedups, and reduces peak memory by 76.6%.

High-Resolution Scaling. We conduct additional resolution-scaling experiments with square padding. Each entry reports time (ms) / peak memory (GiB).

Method1536×15362048×20483072×30724096×4096
UltraMatch, R=6R=643.5 / 0.5974.7 / 0.97171.5 / 2.19333.8 / 4.22
UltraMatch norm filtering, α=0.65\alpha=0.6561.0 / 6.06132.1 / 18.28OOMOOM
UltraMatch dense268.8 / 20.54OOMOOMOOM
ELoFTR dense592.1 / 21.17OOMOOMOOM
ELoFTR + routing333.8 / 6.00638.3 / 10.411898.3 / 15.78OOM
ELoFTR norm filtering, α=0.65\alpha=0.65367.3 / 10.56OOMOOMOOM

At 2048×2048, UltraMatch routing requires 0.97 GiB, compared with 18.28 GiB for norm filtering, and continues to run at 4096×4096. For ELoFTR, routing runs at 3072×3072, while dense matching and the tested norm-filtering setting run out of memory at 2048×2048.

AuthorsOpenReview ↗

Response to Reviewer NQGU (2/3): Low-overlap robustness and normalization

W2 / Q2. Reliability on Low-Overlap Pairs

Overlap Groups and Coverage Definitions. We group MegaDepth-1500 using the precomputed GT co-visibility scores associated with the test pairs. The 0.1–0.3, 0.3–0.5, and 0.5–0.7 groups contain 600, 600, and 300 pairs, respectively.

Overlap 0.1–0.3. We compare dense inference with different routing sizes within UltraMatch. We report GT coverage as both the mean per-pair retention rate and the pooled retention rate across all GT correspondences. Final correct-match count and spatial coverage describe the output correspondences separately from candidate retention.

Routing size / modeAUC@5 / @10 / @20 (%) ↑Correct matches / pair ↑Precision (%) ↑Correct-match spatial coverage (%) ↑GT coverage, per-pair / pooled (%) ↑
Dense63.2 / 77.9 / 87.6295096.5140.08100.00 / 100.00
R=1R=160.0 / 75.4 / 85.4282590.8542.1673.45 / 79.46
R=4R=460.5 / 76.0 / 85.9307993.8942.1592.61 / 96.84
R=6R=662.2 / 77.1 / 86.7307294.6041.8995.10 / 98.18
R=8R=861.1 / 76.8 / 86.8306494.9941.7296.38 / 98.75
R=16R=1662.5 / 76.8 / 86.6304495.6641.3198.31 / 99.44

On the 0.1–0.3 subset, R=6R=6 retains 98.18% of pooled GT correspondences (95.10% mean per-pair coverage) and yields more correct matches and broader spatial coverage than dense inference, with lower precision and 0.8–1.0 percentage points lower AUC. At R=1R=1, mean per-pair GT coverage drops to 73.45%.

Comparison across Overlap Groups. The following table reports AUC@10 / GT coverage (%) across all overlap groups, with coverage averaged over image pairs.

Routing size / modeOverlap 0.1–0.3, N=600N=600Overlap 0.3–0.5, N=600N=600Overlap 0.5–0.7, N=300N=300
Dense77.9 / 100.0074.3 / 100.0057.8 / 100.00
R=1R=175.4 / 73.4573.5 / 80.0857.2 / 81.73
R=6R=677.1 / 95.1075.4 / 98.1958.0 / 98.80
R=8R=876.8 / 96.3873.5 / 98.7459.3 / 99.30

The lower candidate retention in the 0.1–0.3 group confirms that low overlap places greater demands on a fixed routing size. These grouped results make that limitation explicit.


Q1. Why Higher GT Coverage Does Not Guarantee Higher Pose AUC

Increasing RR improves GT coverage but also broadens competition in Sparse Global Dual-Softmax. We fix features, raw similarities, and eligible output candidates at R=6R=6 and vary only the normalization range.

Normalization rangeMean DS score of the same GT-correct correspondencesCorrect matches / pairCorrect-match spatial coverage (%)
R=6R=60.36873898.646.27
R=8R=80.36693877.345.97

These results support a possible mechanism: broader normalization changes confidence and final selection, so higher candidate coverage need not improve pose AUC.

AuthorsOpenReview ↗

Response to Reviewer NQGU (3/3): Retraining protocol, localization and model release

W3. Retraining Protocol and the Accuracy Gap

Training Settings. All models are trained at 832×832 for 30 epochs, using each architecture's optimizer and schedule. Global batch sizes are 8 for ELoFTR, 16 for JamMa, 4 for SLiM, and 32 for EDM. Routing is inserted only at coarse matching, with R∈{4,6,8}R\in\{4,6,8\} during training, R=6R=6 at inference, a one-token halo, routing temperature τr=0.1\tau_r=0.1, and routing loss weight 0.1. All other settings follow the default configuration of each model.

Checkpoint Selection. For each model in Table 3, we select the checkpoint with the best test performance among those saved during the final four training epochs and use it for the reported results.

Interpreting Tables 1 and 3. EDM's 57.5 → 55.4 AUC@5 difference is between the main benchmark result and the separately retrained dense baseline. This gap is already present before adding routing. Within the matched Table 3 experiment, routing changes EDM's AUC@5 / @10 from 55.4 / 71.2 to 56.1 / 72.1. Table 3 therefore measures the effect of adding routing under a shared training protocol. We trained all models in Table 3 using the same hardware configuration, ensuring a fair comparison under this protocol.


Q3. InLoc DUC2 Correspondence Analysis

Across 313 queries with DUC2 references, UltraMatch produces more PnP inliers but slightly lower mean spatial coverage under identical retrieval and localization settings.

MethodMean PnP inliersMean coverage (%)Expected coverage at 200 unique locations (%)
UltraMatch598.2313.1320.70
ELoFTR492.9913.8121.05

Coverage uses a 16×16 grid. The last column reports expected coverage from 200 uniformly sampled unique inlier locations over 130 queries eligible for both methods; ELoFTR's advantage is 0.35 percentage points.

Selected examples show broader support for ELoFTR in corridor IMG_0967 and UltraMatch in meeting room IMG_1102. These diagnostics concern DUC2 reference images.


Q4. Code and Pretrained Model Release

Yes. We plan to release the code and pretrained models, and we are currently organizing the repository for public release.

Thank you again for your valuable feedback. We welcome any further questions and discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions in Response to Reviewer Comments

We thank all reviewers for their constructive comments. The revised manuscript incorporates selected experiments and clarifications from our responses.

Reviewer NQGU

  • Comparisons of routing, SLiM-inspired norm filtering, and dense matching in UltraMatch and ELoFTR, together with a pooled Top-R baseline for UltraMatch (Sec. 4.5, Table 5).

Reviewer kPQ1

  • Comparisons with simpler sparsification strategies, reporting pose accuracy, runtime, and peak memory (Sec. 4.5, Table 5).
  • ETH3D reconstruction evaluation of UltraMatch at R = 1, 6, and 8 against EDM, reporting reconstruction quality, matching and full-pipeline runtime, and peak process memory (App. F, Table 9).

Reviewer PgGs

  • Comparisons with pooled Top-R selection, norm filtering, and dense matching to assess the benefit beyond simpler candidate-selection strategies (Sec. 4.5, Table 5).
  • Clarification of the identity alignment operation under the default block size, the flattened tensor shape, and the normalization dimension (Sec. 3.6, App. A).
  • Streaming tile-size sensitivity experiments at 1152×1152 and 2048×2048, distinguishing implementation overhead from the reduction in token-pair computation (App. G.2, Table 12).
  • Qualification of the OOM statements to refer to the official implementations of other semi-dense matchers (Abstract, Sec. 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.