acceptodds
Under review as a conference paper at ICLR 2027

Tailor-Made for One: Learning from Each Test Pair for Zero-Shot Cross-Modal Homography Estimation

Abstract

Cross-modal homography methods typically rely on training data specific to the cross-modal combination encountered at deployment. Existing zero-shot methods alleviate this dependency by learning modality-invariant structural representations, but still treat each test pair as a passive input, leaving its pair-specific information unexplored. We introduce Tailor-Made for One (TMO), a zero-shot framework that turns each unlabeled test pair into an adaptation signal. The central challenge is that direct reconstruction-based adaptation can conflate appearance differences with geometric misalignment, allowing modality translation to distort the scene structure. TMO addresses this ambiguity by synthesizing prediction-centered training pairs with varying source geometries around the current alignment and requiring adaptation to remain effective across these pairs. This provides pair-specific supervision that refines appearance transfer while preserving its compatibility with a frozen geometric prior. On the three benchmarks shared with prior zero-shot work, TMO achieves state-of-the-art performance before adaptation. Across six benchmarks, adaptation tailored to each test pair further reduces MACE by 16.9%–50.8%. These results show that an unlabeled test pair is not merely an input to a transferable model, but a source of supervision for zero-shot geometric estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
cross-modal homography estimation, test-time adaptation, zero-shot learning, multimodal image alignment, geometric correspondence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6V3NOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a zero-shot cross-modal homography estimation framework called TMO. Its core innovation lies in using test-time adaptation (TTA) to bridge the gap between pre-training data and real-world data. Experimental results across six benchmark datasets demonstrate consistent performance improvements, with the MACE metric reduced by 16.9%–50.8% following adaptive adjustment.

Critical strengths
  1. The motivation is strong. Training data simply doesn't match real-world data, and this gap is most evident in zero-shot scenarios.

  2. Prediction-centric data synthesis offers a powerful approach for generating training data. It enables appearance adaptation while minimizing geometric distortion, requiring only a single pair of unaligned samples (lacking ground truth). Ablation studies and experiments presented in the appendix further demonstrate its effectiveness.

  3. The results demonstrate strong convergence across all six datasets. Ablation studies further confirm that this paradigm generalizes well across various network architectures.

Critical weaknesses
  1. Certain ablation studies are missing. The paper lacks ablation studies for several parameters. For instance, the sampling range for perturbations, the impact of different LoRA ranks, and the weighting of the loss function.

  2. Some baseline comparisons are missing. The metrics for CCNet are cited directly from existing results rather than being re-evaluated under the experimental setup used in this paper; please verify whether the corresponding code has been made publicly available during the rebuttal phase to ensure a fair comparison. Furthermore, if any Test-Time Adaptation (TTA) methods exist within this field, they should also be included in the baseline comparisons.

  3. The paper lacks a rationale for updating only the modality-transfer network while keeping the homography network frozen. Please provide quantitative experiments to justify this design choice.

  4. The paper lacks evaluations on larger-scale models. It remains unclear whether TMO remains effective when paired with a homography estimator that has larger capacity and stronger baseline generalization.

Questions for authors

Could the authors provide additional ablation studies on key hyperparameters, clarify the fairness of the CCNet comparison and the availability of relevant TTA baselines, and offer quantitative evidence supporting the decision to freeze the homography network? Additionally, does TMO remain effective when applied to larger-capacity homography estimators with stronger baseline generalization? The idea behind this paper is interesting; however, if the authors cannot provide sufficient justification and experimental evidence to address these concerns, I would consider lowering my score.

4 comments
AuthorsOpenReview ↗

Part 1: Hyperparameter Ablations and Additional Baseline Comparisons

Dear Reviewer 6V3N, thank you for recognizing our motivation, prediction-centered synthesis, and consistent gains. This response addresses W1 (hyperparameters) and W2 (baseline comparisons).

W1. Certain ablation studies are missing. The paper lacks ablation studies for several parameters. For instance, the sampling range for perturbations, the impact of different LoRA ranks, and the weighting of the loss function.

Hyperparameter ablations. Table R1 varies perturbation range, LoRA rank, and image/feature loss weights around the baseline: ±24 px, rank 8, and weights (1, 1.5). Each loss sweep fixes the other weight; the shared baseline is listed once. MACE is rounded to 0.01 px from the reported relative changes.

Table R1. Hyperparameter ablations (MACE ↓).

SettingValueOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
Range±88.866.221.791.090.524.23
Range±167.896.511.771.070.424.17
Range±24 (base)8.006.101.621.080.694.09
Range±328.096.032.141.080.484.34
Rank48.056.732.081.090.454.22
Rank8 (base)8.006.101.621.080.694.09
Rank167.785.991.821.060.504.09
Loss weights(1, 0.5)7.735.741.961.080.564.25
Loss weights(1, 1)7.975.811.691.080.694.07
Loss weights(1, 1.5) (base)8.006.101.621.080.694.09
Loss weights(1, 2)7.676.591.941.090.494.25
Loss weights(0.5, 1.5)7.896.551.771.080.584.32
Loss weights(1.5, 1.5)7.725.731.841.070.694.41
Loss weights(2, 1.5)7.685.781.921.070.624.28

Conclusion. Higher LoRA rank can help but may increase compute; ±8 perturbations hurt five datasets relative to ±24. Different datasets favor different loss weights, yet all tested settings improve over unadapted Translator+MCNet on all six datasets. These variations do not alter our core finding: adaptation on a single test pair without ground truth can improve homography estimation.

W2. Some baseline comparisons are missing. The metrics for CCNet are cited directly from existing results rather than being re-evaluated under the experimental setup used in this paper; please verify whether the corresponding code has been made publicly available during the rebuttal phase to ensure a fair comparison. Furthermore, if any Test-Time Adaptation (TTA) methods exist within this field, they should also be included in the baseline comparisons.

To our knowledge, CCNet's homography code has not been publicly released, so we use its published results.

Additional TTA baselines. We surveyed registration and TTA studies and selected Reg-TTR and SGTTA as the most suitable baselines to adapt. Reg-TTR [1] refines a predicted warp using NCC/SSIM and smoothness; our Reg-TTR-H optimizes an eight-DoF homography with MIND-like [3] and HOG [4] losses (weights 1/0.5), capturing modality-independent local structure and gradient orientations, respectively. SGTTA [2] uses feature-statistics, HOG, and model-history guidance; our SGTTA-H updates per-pair MCNet channel scale/bias with HOG [4]/MIND-like [3] losses (2/0.5) and weak statistics (0.01). We tried multiple configurations, selected each method's best on synthetic COCO, and report 300-step MACE on all 5,731 pairs (Table R2). TMO leads on all six datasets.

Table R2. Homography estimation before and after adaptation (MACE; lower is better).

MethodOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
MCNet (no TTA)13.969.225.141.251.315.96
MCNet + Reg-TTR-H (adapted)13.798.964.651.281.285.79
MCNet + SGTTA-H (adapted)12.958.404.261.221.235.42
Translator+MCNet (no TTA)13.668.633.291.301.365.85
Translator+MCNet + TTA (TMO)8.006.101.621.080.694.09

References

[1] Chen et al. Reg-TTR, Test-Time Refinement for Fast, Robust and Accurate Image Registration. arXiv, 2026.

[2] Zhou et al. Test-Time Adaptation via Style and Structure Guidance for Histological Image Registration. AAAI, 2024.

[3] Heinrich et al. MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration. Medical Image Analysis, 2012.

[4] Dalal and Triggs. Histograms of Oriented Gradients for Human Detection. CVPR, 2005.

AuthorsOpenReview ↗

Part 2: Estimator Freezing and Larger-Capacity Models

Dear Reviewer 6V3N, thank you for your review. This response addresses W3 (estimator freezing), W4 (model capacity), and your question.

W3. The paper lacks a rationale for updating only the modality-transfer network while keeping the homography network frozen. Please provide quantitative experiments to justify this design choice.

Why freeze the homography estimator? Pretraining aims to learn modality-invariant alignment; uncertain test-time supervision may disrupt it. Translator adaptation changes appearance while preserving this alignment. Table R3 compares five updates from the same baseline: estimator-only updates give mixed results, while joint updates improve all six. Translator-only TMO beats joint updates on five and trails by 0.01 px on RGB-NIR.

Table R3. Updating different components of Translator+MCNet (MACE ↓).

UpdateOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
Translator+MCNet (no TTA)13.668.633.291.301.365.85
FNet only13.528.563.411.271.265.86
Homography head only13.588.933.631.221.376.15
Full homography estimator13.739.203.591.221.136.05
Homography estimator + translator LoRA8.237.511.711.130.685.61
Translator LoRA only (TMO)8.006.101.621.080.694.09

W4. The paper lacks evaluations on larger-scale models. It remains unclear whether TMO remains effective when paired with a homography estimator that has larger capacity and stronger baseline generalization.

Larger-capacity estimator. We retain RoMa's full backbone [1], add a homography head, and train on MS COCO. We pair this estimator with the translator in TMO, adapting only translator LoRA while freezing the estimator. Standard RoMa has about 111M parameters [2] versus MCNet's 0.85M [3] (about 130×). Unadapted Translator+RoMa beats Translator+MCNet on four datasets; 300-step TMO lowers MACE on all six (Table R4). This supports TMO's applicability to a larger estimator while preserving its pretrained alignment function.

Table R4. Translator+RoMa under TMO before adaptation (RoMa$0$) and after 300 steps (RoMa${300}$), with only translator LoRA updated (MACE ↓).

MethodOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
RoMa0\mathrm{RoMa}_{0}8.074.422.541.731.593.63
RoMa300\mathrm{RoMa}_{300}6.203.861.851.551.033.22

Q. Questions for Authors. Could the authors provide additional ablation studies on key hyperparameters, clarify the fairness of the CCNet comparison and the availability of relevant TTA baselines, and offer quantitative evidence supporting the decision to freeze the homography network? Additionally, does TMO remain effective when applied to larger-capacity homography estimators with stronger baseline generalization? The idea behind this paper is interesting; however, if the authors cannot provide sufficient justification and experimental evidence to address these concerns, I would consider lowering my score.

Please see W1–W2 in our first response for hyperparameters and baselines, and W3–W4 above for the estimator and capacity experiments.

References

[1] Edstedt et al. RoMa: Robust Dense Feature Matching. CVPR, 2024.

[2] Lu and Du. Raising the Ceiling: Conflict-Free Local Feature Matching with Dynamic View Switching. ECCV, 2024.

[3] Zhu et al. MCNet: Rethinking the Core Ingredients for Accurate and Efficient Homography Estimation. CVPR, 2024.

Reviewer_6V3NOpenReview ↗

Response to Authors' Rebuttal

Thank you for your detailed response. The rebuttal has thoroughly addressed all of my concerns. Accordingly, I have raised my score to 4.

AuthorsOpenReview ↗

Response to Reviewer 6V3N

We sincerely appreciate your positive reassessment and for raising your score. Your insightful comments have been tremendously helpful in refining our framing, and we are pleased that our rebuttal addressed your questions. We will carefully revise the manuscript according to your advice to make our presentation even clearer and more rigorous. Thank you once again for your dedication and thoughtful review.

Reviewer_FaYkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a zero-shot cross-modal homography estimation framework that adapts to individual unseen test pairs. The framework consists of a modality translator and a homography estimator, both pretrained on synthetic image pairs. At test time, the homography estimator remains frozen, while the translator is adapted to the appearance differences of each test pair using LoRA. To prevent the translator from absorbing residual geometric errors, the authors introduce prediction-centered data synthesis, which generates training pairs with varying source geometries around the current homography prediction. Both pixel-level reconstruction and feature alignment losses are employed during pretraining and adaptation. The adapted LoRA parameters are discarded before processing the next test pair. Experiments on six benchmarks demonstrate consistent improvements through pair-specific adaptation.

Critical strengths
  1. The main strength of this paper is its test-time adaptation strategy for zero-shot cross-modal homography estimation. By separating modality translation from homography estimation and introducing prediction-centered data synthesis, the framework encourages appearance adaptation while reducing the risk of absorbing residual geometric errors. This addresses an important challenge in adapting to unseen modality combinations without geometric supervision.

  2. The paper provides convincing experimental evidence supporting the proposed adaptation strategy. In particular, Appendix evaluates the geometric equivariance of the translator by comparing translation before and after geometric warping. The results demonstrate that prediction-centered perturbations substantially improve geometric consistency, providing empirical support for the proposed method.

Critical weaknesses
  1. The method section is difficult to follow due to the large number of variables and transformations introduced without a clear problem formulation. A dedicated formulation subsection defining the source and target images, homography conventions, and optimization objectives would improve readability. In addition, the homography encoder f_\phi, which plays an important role in feature supervision, is only briefly mentioned without sufficient architectural details. Clarifying its relationship with the complete homography estimator F_\phi would improve reproducibility.

  2. The proposed framework requires iterative optimization for each test pair, with 300 adaptation steps used in the main experiments. Since the adapted LoRA parameters are discarded after each pair, this optimization cost must be incurred repeatedly. The computational overhead and accuracy–efficiency tradeoff are not sufficiently characterized in the main results, raising concerns about practical scalability.

Questions for authors
  1. Could the authors clarify the architecture of the homography encoder f_\phi used in Eq. (6)? Specifically, which feature layers of F_\phi are used, and are these features extracted before or after cross-image interaction?

  2. Could the authors report the average adaptation time per test pair and the computational overhead relative to standard inference without adaptation? Understanding the accuracy–efficiency trade-off would help assess the practical applicability of the proposed framework.

3 comments
AuthorsOpenReview ↗

Clarifications on Formulation, Encoder Features, and Computational Cost

Dear Reviewer FaYk, thank you for your thoughtful feedback and for recognizing our adaptation strategy and geometric-consistency analysis. We address W1/Q1 (formulation and encoder details) and W2/Q2 (cost and the accuracy–efficiency tradeoff).

W1/Q1 (summary). Clarify the problem formulation, homography convention, optimization objectives, and the relationship between fϕf_\phi and FϕF_\phi, including which feature layers are used and whether they precede cross-image interaction.

Formulation and notation. We have revised Sections 3.1–3.3 to define the variables and transformation sequence explicitly. Given source XsX_s and target XtX_t, the translator produces Z=Gθ,ψ(Xs,Xt)Z=G_{\theta,\psi}(X_s,X_t) to match target appearance while retaining source coordinates. The complete estimator Fϕ(Z,Xt)F_\phi(Z,X_t) predicts four-corner displacements, which DLT converts into a source-to-target homography. Our warp operator takes an input-to-output coordinate map and implements it by inverse sampling. We also clarify the order of the current alignment, residual perturbation, and correction warp.

Encoder and feature selection. fϕℓf_\phi^\ell denotes the feature-extraction mapping at level ℓ\ell within FϕF_\phi, not an additional network. For MCNet, Eq. (6) uses all three levels of the encoder-output feature pyramid (L=3L=3), extracted independently from each image before cross-image interaction. These are the encoder features used by the homography prediction head. We choose them because they directly support geometric estimation, providing task-relevant supervision for the translator. This rule fixes the feature levels for each estimator across datasets and test pairs, making the selection reproducible.

Objectives and gradient flow. Pretraining uses known corner displacements to supervise the estimator, and reconstruction plus cosine feature losses to train the translator. Eq. (6) compares each translated output with its geometrically aligned synthetic target at corresponding feature levels. The encoder parameters remain frozen, but gradients through the translated-image branch update the translator. At test time, the correction warp aligns the translated source with the observed target; both images are masked before feature extraction. Both pretrained backbones remain frozen, and only translator LoRA is updated using reconstruction and feature losses, without ground-truth geometry. The correction is detached during each update, while the anchor estimate is refreshed as the translator changes.

W2/Q2 (summary). Report per-pair adaptation time and computational overhead relative to standard inference, and characterize the accuracy–efficiency tradeoff.

Computational cost. Measurements use an RTX 4090 and 128×128 inputs, with BF16 for the translator and FP32 for the homography estimator. Approximate FLOPs include backpropagation. At 300 steps, TMO takes 10.452 s/pair with 0.932 GB peak GPU memory, compared with 0.029 s and 0.063 GB for MCNet alone. Translator+MCNet without adaptation takes 0.078 s (Table R1). This cost recurs per pair because LoRA parameters are reset.

Table R1. Per-pair computational cost; FLOPs are approximate.

MethodStepsTime/pair (s) ↓Peak GPU memory (GB) ↓Approx. FLOPs/pair (G) ↓
MCNet only00.0290.0639.79
Translator+MCNet (TMO$_0$)00.0780.09217.14
TMO502.7580.9322,356.69
TMO1004.4510.9324,696.24
TMO30010.4520.93214,054.44

Cost in context and tradeoff. TTA typically incurs additional computational cost and runtime [1,2]. TMO already improves all six datasets at 50 steps (2.758 s/pair); 100 steps (4.451 s/pair) achieve 57.5%–76.7% of the 300-step MACE reduction. At 300 steps (10.452 s/pair), accuracy improves further. Thus, the budget can be chosen according to runtime and accuracy requirements. Table R2 uses TMO checkpoints from paper Table 2 and our estimator-only control.

Table R2. Accuracy–efficiency tradeoff: time per pair versus MACE (pixels) on six datasets; lower is better.

MethodStepsTime/pair (s) ↓OPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
MCNet only00.02913.969.225.141.251.315.96
Translator+MCNet00.07813.668.633.291.301.365.85
TMO502.75810.677.152.671.250.955.06
TMO1004.4519.426.692.331.170.874.57
TMO30010.4528.006.101.621.080.694.09

References

[1] Zhang et al. MEMO: Test Time Robustness via Adaptation and Augmentation. NeurIPS, 2022.

[2] Gandelsman et al. Test-Time Training with Masked Autoencoders. NeurIPS, 2022.

Thank you for your responses, and they have addressed all my concerns. I keep the week accept decision.

AuthorsOpenReview ↗

Response to Reviewer FaYk

Thank you for your careful review of our work. We are pleased that our responses have addressed your concerns, and we truly appreciate your suggestions. We will revise the manuscript to incorporate your feedback and further improve its clarity and rigor. Thank you again for your valuable time and effort.

Reviewer_fwwHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TMO, a zero-shot cross-modal homography method that adapts to each test pair at inference time. A conditional modality translator and a homography estimator (MCNet by default) are pretrained only on MS COCO, using synthetic appearance changes and random four-point warps as homography generation. At test time, only LoRA adapters in the translator are optimized for 300 steps per pair.

Each step does the following:

  1. warps the source by the current estimate with and added random residual,
  2. translates it toward the target appearance,
  3. aligns it with a correction from the frozen homography estimator,
  4. minimizes a masked L1 reconstruction loss plus a cosine loss on frozen estimator features, with only LoRA parameters updated. The anchor estimate is then refreshed.

On six benchmarks, adaptation lowers MACE by 16.9-50.8% relative to the pretrained model without test-time adaptation (TMO_0). TMO_0 itself is better than the reported CCNet numbers on the three shared datasets. Ablations cover the choice of estimator and translator, the two losses, and the perturbation strategy.

Critical strengths
  1. The failure mode is well identified. Reconstructing against a single alignment that can be possibly wrong can let the translator absorb residual geometric error. Sampling residual warps around the current prediction is a simple remedy: Table 4 shows a clear gain over the no-residual variant (9.52 to 8.00 on OPT-SAR, 8.21 to 6.10 on DPDN for example). Tables 8-9 in the appendix also test the mechanism. Without residuals, warp inconsistency rises above its starting value, from 1.94 to 2.42 pixels, whereas it drops to 0.75 pixels with residuals, and 96.5% of pairs improve.

  2. The gains are consistent and the evaluation is careful. Adaptation helps on all six benchmarks, with three homography estimators and two translators (Table 3). The gains seem consistent across the different benchmarks, and even pairs with good initial estimates near 1 pixel improve, which suggests the adaptation signal is helpful.

Critical weaknesses
  1. The paper does not report the cost, and does not show that per-pair learning is needed. Each test pair needs 300 AdamW steps through the translator, plus several estimator passes per step. Every baseline in Table 1 uses one forward pass, yet no runtime, memory or FLOPs comparison is given. Appendix C discusses cost only qualitatively. Also, one natural control is missing, optimize the homography directly against the same frozen-feature loss, with no translator. Table 4 makes this control important:
  • Feature-only adaptation is within 0.15 pixels of the full objective on five of six datasets.
  • Only DPDN clearly benefits from adding reconstruction.

So it seems much of the gain comes from feature alignment in the frozen encoder rather than from learning the appearance relationship. Also, this asks the question on whether the translator is the good place to adapt with LoRA.

  1. The zero-shot comparison is thin.
  • There is no estimator only row (MCNet pretrained on the same dataset, with the same augmentations, without the translator), so the translator's contribution before adaptation is unknown.
  • The generalizable matchers discussed in Related Work (ZeroCo, MINIMA, GIM) could be run with RANSAC on the same evaluations, and could add more zero-shot methods results.

Overall, I think the idea and the mechanism analysis interesting. However, I cannot tell how much of the gain is specific to per-pair translator learning. I currently lean towards rejection, but I would raise my score if the authors report the cost and the direct optimization and estimator-only controls.

Questions for authors
  1. What are the time and peak GPU memory per pair for 300 steps, compared to a single MCNet forward pass?
  2. What fraction of pairs end with higher error after adaptation, especially on OPT-SAR and DPDN, where the first estimate is often poor?
  3. What MACE does direct optimization of the homography under L_feat achieve, using the same frozen encoder, budget, and estimate refresh but no translator?

Small mistake: Figure 3, for the pair on the bottom right line B, I think it should be Z_0, ... instead of C_0, ...

5 comments
AuthorsOpenReview ↗

Part 1: Computational Cost and the Role of Translator Adaptation

Dear Reviewer fwwH, thank you for your thoughtful feedback. We hope our clarifications and additional experiments address your concerns. Below, we address W1/Q1 on cost and translator adaptation.

W1 (summary). Report runtime, memory and FLOPs; establish the benefit of per-pair learning using a direct-homography control without a translator; explain the strong feature-only results and why translator LoRA is the appropriate adaptation target.

Computational cost. Measurements use an RTX 4090 and 128×128 inputs, with BF16 for the translator and FP32 for the homography estimator. Approximate FLOPs include backpropagation. At 300 steps, TMO takes 10.452 s/pair with 0.932 GB peak GPU memory, versus 0.029 s and 0.063 GB for a single MCNet forward pass. Translator+MCNet without adaptation takes 0.078 s. This per-pair overhead recurs because adapters are reset for each new pair.

Table R1. Per-pair computational cost; FLOPs are approximate.

MethodStepsTime/pair (s) ↓Peak GPU memory (GB) ↓Approx. FLOPs/pair (G) ↓
MCNet only00.0290.0639.79
Translator+MCNet (TMO$_0$)00.0780.09217.14
TMO502.7580.9322,356.69
TMO1004.4510.9324,696.24
TMO30010.4520.93214,054.44

Cost in context and tradeoff. TTA typically incurs additional computational cost and runtime [1,2]. TMO already improves all six datasets at 50 steps (2.758 s/pair); 100 steps (4.451 s/pair) achieve 57.5%–76.7% of the 300-step MACE reduction. At 300 steps (10.452 s/pair), accuracy improves further. Thus, the budget can be chosen according to runtime and accuracy requirements.

Table R2. Accuracy at different adaptation budgets (MACE in pixels ↓).

MethodStepsOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
MCNet only013.969.225.141.251.315.96
Translator+MCNet013.668.633.291.301.365.85
TMO5010.677.152.671.250.955.06
TMO1009.426.692.331.170.874.57
TMO3008.006.101.621.080.694.09

Direct optimization without a translator. Following your suggestion, we directly optimized the homography using the frozen encoder and cosine feature discrepancy for 300 steps, without a translator. We initialize HH from the pretrained estimator's prediction and extract source and target features with the frozen encoder. At each step, the current HH differentiably warps the source features into the target coordinate frame, and we compute the cosine discrepancy between the warped source features and the target features. Gradients propagate through the differentiable warp to update the homography parameters directly, while all network parameters remain frozen. Thus, the optimization variable is the homography itself. Table R5 in our other response reports the results: MACE increases on five datasets, including DPDN (9.22 to 10.91) and Google Map (5.14 to 6.66), while OPT-SAR improves by only 0.03 px (13.96 to 13.93).

We also tested feature-loss adaptation of FNet only, the homography head only, and the full estimator without a translator. These increase MACE on five, five, and six datasets, respectively. In contrast, translator LoRA adaptation improves all six datasets over its own pretrained initialization (Table R2). These controls support the contribution of translator adaptation: optimizing the homography or estimator with feature loss alone does not reproduce its gains under the tested settings.

Roles of feature and reconstruction losses. We agree that feature supervision accounts for most of the adaptation gain on five datasets, while reconstruction provides a clearer additional benefit on DPDN (paper Table 4). The feature-only variant still updates translator LoRA through the frozen encoder; its strong performance therefore supports the effectiveness of feature-guided translator adaptation.

The two losses supervise the translator at different levels: feature loss encourages agreement in the estimator's representation, while reconstruction provides pixel-level appearance supervision. Their relative contributions vary across datasets, as also reflected in our additional loss-weight ablations (Table R1 in our response to Reviewer 6V3N). For example, with the reconstruction weight fixed at 1, reducing the feature weight from 1.5 to 0.5 improves DPDN from 6.10 to 5.74, but worsens Google Map from 1.62 to 1.96. These results motivate retaining a weighted combination of both signals, while acknowledging that reconstruction adds only modest gains on most datasets. Our main results use the same weights, (1, 1.5), across all six datasets.

Q1. What are the time and peak GPU memory per pair for 300 steps, compared to a single MCNet forward pass?

Please refer to our response to W1 above.

References

[1] Zhang et al. MEMO: Test Time Robustness via Adaptation and Augmentation. NeurIPS, 2022.

[2] Gandelsman et al. Test-Time Training with Masked Autoencoders. NeurIPS, 2022.

AuthorsOpenReview ↗

Part 2: Zero-Shot Comparisons, Optimization Controls, and Adaptation Outcomes

Dear Reviewer fwwH, thank you for your thoughtful feedback. We hope that our clarifications and additional experiments address your concerns. This response addresses W2, Q2–Q3, and the Figure 3 typo; W1/Q1 are answered separately.

W2. The zero-shot comparison is thin.

RANSAC comparisons. We agree these comparisons are valuable. We tested multiple feature layers and RANSAC [1] settings, reporting each method's best results (Table R3). These methods may use real cross-modal training pairs: MINIMA [3] fine-tunes its infrared generator on M3FD and LLVIP, while our models use COCO only. TMO$0$ leads on five datasets; TMO${300}$ leads on all six.

Table R3. Generalizable matchers with RANSAC versus TMO before and after 300 adaptation steps (MACE ↓).

MethodOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
ZeroCo-L [2]22.0025.8117.125.217.6116.47
MINIMA-RoMa [3]27.659.414.721.371.125.91
GIM-LoFTR [4]31.0524.8814.421.893.7029.59
TMO0\mathrm{TMO}_{0}13.668.633.291.301.365.85
TMO300\mathrm{TMO}_{300}8.006.101.621.080.694.09

Estimator-only controls. Table R4 compares MCNet, IHN, and RHWF with/without a translator ("+" denotes with), using identical COCO data and augmentations. The results suggest that translation helps homography estimation under larger modality gaps, such as OPT-SAR and DPDN.

Table R4. Estimators with/without a translator (MACE ↓).

MethodOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
MCNet13.969.225.141.251.315.96
MCNet+13.668.633.291.301.365.85
IHN12.7010.215.591.982.017.99
IHN+12.629.114.621.812.037.70
RHWF14.7411.676.871.822.198.19
RHWF+11.167.823.381.611.605.93

Optimization without a translator. Using cosine feature loss and 300 optimization steps, updating FNet, the homography head, or all of MCNet worsens MACE on five, five, and six datasets, respectively. Directly optimizing HH with MCNet frozen worsens five datasets and improves OPT-SAR by only 0.03 px (Table R5). These controls do not yield consistent improvements.

Table R5. Optimization without a translator (MACE ↓).

DatasetDirect MCNetUpdate FNet onlyUpdate homography head onlyUpdate allDirect HH
OPT-SAR13.9614.7425.3423.7213.93
DPDN9.2211.1012.2918.5110.91
Google Map5.145.437.907.566.66
Google Earth1.251.181.081.981.26
RGB-NIR1.312.321.816.631.32
M3FD5.9611.7416.3817.327.42

Questions For Authors: 2. What fraction of pairs end with higher error after adaptation, especially on OPT-SAR and DPDN, where the first estimate is often poor?

Pairwise TTA outcomes. TTA improves 83.87% of OPT-SAR pairs and 73.91% of DPDN pairs (Table R6).

Table R6. Pairs improved after TTA (% ↑).

DatasetOPT-SARDPDNGoogle MapGoogle EarthRGB-NIRM3FD
Improved after TTA83.87%73.91%89.60%68.39%71.79%69.47%
  1. What MACE does direct optimization of the homography under L_feat achieve, using the same frozen encoder, budget, and estimate refresh but no translator?

Table R5 reports the direct-HH results. Starting from the pretrained estimator's prediction, we treat the homography parameters as the optimization variables. The frozen encoder extracts features from both images; the current HH differentiably warps the source features to align them with the target features. We then minimize their cosine discrepancy by backpropagating through the warp to update HH directly. This process runs for 300 steps, with all network parameters frozen and no translator involved.

Small mistake: Figure 3, for the pair on the bottom right line B, I think it should be Z_0, ... instead of C_0, ...

Thank you for catching this; we corrected Figure 3.

References

[1] Fischler and Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. CACM, 1981.

[2] An et al. Cross-View Completion Models are Zero-shot Correspondence Estimators. CVPR, 2025.

[3] Ren et al. MINIMA: Modality Invariant Image Matching. CVPR, 2025.

[4] Shen et al. GIM: Learning Generalizable Image Matcher from Internet Videos. ICLR, 2024.

Reviewer_fwwHOpenReview ↗

Thank you for the rebuttal

Thank you for the rebuttal, which addresses my main concerns.

You now report the cost and the accuracy at smaller budgets. Please include both in the main paper.

The controls in Table R5 show that optimizing the homography or the estimator with the same feature loss does not reproduce the gains, which supports adapting the translator. The estimator-only control shows that the translator contributes little before adaptation, so I would present it mainly as the adaptation target.

I will raise my score to weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.