- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents UltraMatch, an efficient semi-dense image matching framework. A block-level router uses low-resolution interacted features to select a small set of candidate target regions before token-level coarse matching. Dual-Softmax is then applied over the retained sparse correspondence graph, preserving competition across routed regions. Combined with a reparameterized backbone and a lightweight refinement head, the framework achieves competitive geometric accuracy with substantial runtime and memory savings. The paper also evaluates the routing strategy within other matching architectures and demonstrates inference at high input resolutions.
Critical strengths
- Preserving cross-region matching competition is a meaningful design choice. Source tokens from different blocks may compete for the same target token, so independently normalizing each routed region can lose important matching constraints. The proposed sparse global normalization retains these interactions among the selected candidates. The ablation results support this choice, with AUC@5 decreasing from 57.4 to 54.1 when normalization is performed independently within routed blocks.
- The efficiency improvements are substantial and supported by several experiments. Within the same architecture, replacing routing with dense matching increases runtime from 32.26 ms to 73.78 ms. Transfer experiments with retrained variants of existing matchers provide further evidence that the benefit extends beyond the proposed backbone. The full-FP32 comparison also supports gains beyond mixed-precision optimization, while the high-resolution experiments demonstrate practical scalability.
Critical weaknesses
- The discussion and comparison with existing coarse-matching sparsification approaches are insufficient. Comparisons against dense matching establish that candidate reduction is effective, but do not sufficiently establish the advantage over existing feature filtering or straightforward hierarchical candidate selection. A comparison under comparable training and computational budgets would help isolate the contribution of the proposed routing formulation.
- The conceptual novelty appears limited. The routing stage can be understood as an additional, coarser matching stage preceding token-level matching and refinement. Although the accompanying supervision, halo expansion, and sparse normalization are empirically useful, the paper could more clearly explain what new algorithmic insight the combination provides beyond a conventional hierarchical matching strategy.
- The presentation is occasionally indirect. Some descriptions rely on abstract terminology where concrete operations and tensor dimensions would be more informative.
Questions for authors
Clarification requests:
-
What exactly does implement in Eq. (1)? Please specify the operation and input/output spatial dimensions. With the default , the routing grid appears to have the same spatial resolution as the features. Is an identity operation in this setting, or does it perform additional resampling or alignment?
-
How sensitive is the reported speed advantage to the streaming dense baseline’s implementation? The baseline uses a two-pass, chunked PyTorch implementation. While it provides a useful memory-efficient reference, its latency does not establish that efficient dense implementations necessarily incur comparable overhead. Please clarify the implementation bottlenecks and distinguish the benefit of candidate sparsification from implementation overhead.
-
Could the authors explicitly qualify the approximately 2K OOM claim as applying to the tested official configurations? Table 9 shows that streaming dense matching reduces memory to 0.43 GiB, comparable to routing’s 0.44 GiB at the evaluated resolution. The observed OOM thresholds should therefore be distinguished from limitations intrinsic to the matching algorithms.
Words to authors: I lean toward acceptance given the substantial efficiency gains and supporting ablations, although the novelty and the attribution of the memory savings remain insufficiently clear. Satisfactory answers to the questions above could lead me to raise my recommendation to clear accept.
3 comments
Response to Reviewer PgGs (2/2): Implementation details, streaming and memory limits
W3 / Q1. Concrete Operations and Tensor Dimensions
Thank you for pointing out this ambiguity and the need for more concrete descriptions.
Eq. (1). With the default , the routing grid has spatial dimensions , exactly matching the interacted features . Therefore, : it directly passes these features through, without pooling, interpolation, resampling, or additional alignment. We retain in Eq. (1) solely for generality of the formulation. Both its input and output have shape , where is the batch size and is the preprocessed image size.
The subsequent is a shared Conv–BN projection from 256 to 64 channels, producing without changing spatial resolution. Flattening the spatial dimensions yields , where ; then applies normalization along each descriptor's 64 channels.
We have clarified the identity alignment in Sec. 3.6 and the flattened tensor shape and normalization dimension in Appendix A of the revised manuscript.
Q2. Streaming Implementation Sensitivity and Efficiency Attribution
The 289.22 ms streaming result at 1152×1152 in Table 9 of the original submission (Table 11 in the revision) is specific to our two-pass PyTorch implementation. We additionally vary tile size, retaining complete global Dual-Softmax and two passes over all token similarities. Inputs are resized and square-padded to the listed dimensions.
| Implementation | 1152×1152: time (ms) ↓ | 1152×1152: memory (GiB) ↓ | 2048×2048: time (ms) ↓ | 2048×2048: memory (GiB) ↓ |
|---|---|---|---|---|
| Native dense | 74.16 | 8.26 | OOM | OOM |
| Streaming dense, tile 512 | 990.71 | 0.43 | 9396.98 | 0.94 |
| Streaming dense, tile 1024 | 285.09 | 0.43 | 2415.14 | 0.94 |
| Streaming dense, tile 2048 | 148.96 | 0.43 | 1293.67 | 0.94 |
| Streaming dense, tile 4096 | 136.24 | 0.57 | 1168.80 | 0.94 |
| UltraMatch, | 31.19 | 0.43 | 74.53 | 0.97 |
Implementation Overhead. Increasing the tile size reduces streaming latency from 990.71 to 136.24 ms at 1152×1152, confirming substantial implementation sensitivity. The two-pass implementation recomputes dense similarities, while chunking introduces per-tile reductions and kernel launches. Against the fastest tested streaming setting, routing is 4.37× faster at 1152×1152 and 15.68× faster at 2048×2048.
Candidate Reduction. Both native and streaming dense matching evaluate the complete token-pair space. Routing directly reduces that computation, as quantified by the similarity GFLOPs in our W1 response (Part 1). Independently of streaming, the retrained native-dense comparison in Table 4 gives 73.78 → 32.26 ms (2.29×) at 1152×1152.
Q3. Scope of the OOM Claim
We have qualified the OOM statements in the revised abstract and introduction to refer to the official implementations.
This is an observed implementation limit. Streaming dense matching successfully runs at 2048×2048 with 0.94 GiB, slightly below routing's 0.97 GiB, demonstrating that avoiding full-matrix storage can remove this memory bottleneck. Routing additionally reduces the number of evaluated candidates, achieving 74.53 ms versus 1168.80 ms for the fastest tested streaming configuration. The memory bottlenecks in the tested official implementations can indeed be addressed using additional techniques such as streaming matching. However, our router achieves ultra-low memory consumption and very fast inference simultaneously by reducing candidate computation while avoiding full-matrix storage.
Thank you again for your valuable feedback. We welcome any further questions and discussion.
Thank you for your positive reassessment of our work and for your time and consideration.
Response to Reviewer PgGs (1/2): Sparsification baselines and routing contributions
Thank you for recognizing our scalability and efficiency evaluation. We clarify the comparison with simpler sparsification strategies, the contribution beyond hierarchical search, the concrete routing operations, and the interpretation of the streaming and memory results below. We address your comments in two parts.
W1. Comparison with Simpler Sparsification Strategies
Same-Architecture Comparisons. We compare UltraMatch with pooled-feature block similarity followed by Top-R selection, SLiM-style feature-norm filtering, and dense matching within the UltraMatch architecture on MegaDepth. The pooled baseline uses the same routing size R=6; norm filtering retains tokens on both sides before matching the retained tokens.
Summary. Pooled Top-R provides a useful control with the same candidate set size and nearly identical runtime and memory. Norm filtering reduces dense matching cost, but still requires 16.76–39.46 GFLOPs and 1.92–3.34 GiB, compared with 2.29 GFLOPs and 0.43 GiB for routing. Relative to dense execution, routing reduces token-similarity computation by 96.1×. The retrained ablations below further isolate the contributions of halo expansion and sparse global normalization.
W2. Contribution beyond Hierarchical Search
Our strategy builds on the hierarchical principle of multiscale optimization. We translate this principle into a lightweight Transport Path Router, Sparse Global Dual-Softmax, and halo expansion to overcome the computation and memory bottlenecks of dense coarse matching in semi-dense pipelines.
Routing and Boundary Coverage. The router uses the interacted 1/32 features to select a small set of target blocks for 1/8 token matching. Similarities are evaluated only along these selected paths, avoiding construction of the full token-to-token matrix. A one-token halo extends each selected block to retain valid correspondences near its boundaries with limited additional computation.
Sparse Global Dual-Softmax. Our normalization preserves competition across source blocks while operating only on the routed token pairs, maintaining strong matching performance at low computational cost. Table 4 shows that global normalization substantially improves accuracy over local normalization at essentially unchanged latency. The halo ablation further confirms the benefit of preserving boundary correspondences.