acceptodds
Withdrawn from ICLR 2027

Seg2Reg: Structure-Preserving Appearance Randomization for Multimodal Retinal Image Registration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal retinal image registration (MRIR) aims to precisely align retinal vascular structures across modalities, supporting clinical diagnosis and treatment planning. However, the development of MRIR is hindered by the scarcity of large-scale annotated multimodal retinal image pairs. To address this issue, we propose a Seg2Reg learning framework that employs Retinal Structure-Preserving Appearance Randomization (RSPAR) to construct a synthetic registration dataset from retinal vessel segmentation data. RSPAR generates image pairs with randomized appearances while preserving vascular structure and exact geometric correspondences. The preserved vascular structures and exact geometric correspondences provide supervision for vessel-aware keypoint detection and feature matching, respectively, enabling registration learning without manually registered pairs. Beyond the limited training data, strong cross-modal appearance variations can cause anatomically corresponding locations to exhibit substantially different local features, leading to ambiguous feature matching. Motivated by this observation, we introduce Retinal Vascular Geometric Consistency (RVGC) to characterize whether candidate correspondences agree with the global retinal vascular geometry, and develop an RVGC Module for learned reliability-guided correspondence filtering and geometric estimation. Extensive experiments on the public CFFA benchmark and the private MICRO dataset demonstrate that Seg2Reg achieves state-of-the-art multimodal retinal image registration performance among the evaluated methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal retinal image registration, synthetic supervision, appearance randomization, feature matching, geometric consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m4aGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main component of this paper is to use vessel-map-generated synthetic pairs to train retinal keypoint detection and correspondence establishment models. The high-level idea is interesting but the detailed technical design is moderately novel and is not well justified. The proposed retinal-constrained training, vascular feature attention and Support-Aware Polynomial refinement are technically concerning. Experiments are moderately convincing and lack rigorous comparison.

Critical strengths
  1. Putting aside the basic augmentation-based methods used for synthetic pairs and lack of theoretical justification of how this handles the appearance gap at detection with real modalities. The idea to generate synthetic pairs from vessel maps seems novel and is interesting.

  2. The size of the test set is good (over 1000 pairs) for MRIR. But the opacity of the dataset then weakens this strength.

Critical weaknesses
  1. The RSPAR is mainly empirically justified by the final registration performance, which is not convincing enough. There's no theoretical justification of why the vessel-map synthesized (using basic augmentation methods) image-trained models can handle the domain appearance difference with the real modalities. The author also didn't provide any direct ablation nor simple demonstration on the detected keypoints qualities, e.g., whether they reflect real anatomical structures such as vessel bifurcations, which are useful for registration.

  2. The vessel loss in RSPAR encourages detecting keypoints on vessels, however, not all locations on the vessels are useful for retinal matching and registration. The method doesn't handle this issue nor explore and compare options which further encourage the keypoints to be detected on structurally more distinctive locations.

  3. The proposed Vascular Geometry-Aware Attention is not fundamentally different from the attention mechanism commonly used in image correspondence models such as SuperGlue, which lacks novelty. The name "Vascular Vascular Geometry-Aware Attention" is also overselling the method. The only thing about vessels is that the inputs are keypoints that are encouraged to be detected from the vessel. The formation of the descriptor fed into the RVCG doesn't have any adaptation to vessel features, nor does the model design reflect unique thinking on "how we should address and match features from retinal vessels".

  4. Superpoint-Lightglue established sparse correspondence doesn't provide exact sub-pixel level precision between the source and target keypoints, so in RGE, the control point locations can be inaccurate and so is the computed delta. Additionally, even with the blocky strategy, the computation is also largely depending on the sparsity of the keypoints around. Together this can easily cause local distortion in the registration. The author didn't discuss regularization methods to handle this issue. I also don't see any qualitative result on the final registered image in both the main paper and the appendix, which makes the registration quality dubious. Furthermore, since this topic is about deformable residual alignment, the author also needs to compare with the dense corresondence-based retinal registration baselines.

  5. The author didn't compare with vessel segmentation-based registration methods, which generally shows better performance for MRIR than directly establishing correspondence from images.

  6. Annotated retinal vessel map is scarce, the RSPAR is fundamentally limited by this constraint, which prevents extending this method to larger training dataset to achieve higher generalization, robustness and accuracy.

  7. The details of the main test set (MICRO) is largely unavailable (what modalities and pairs, number of images for each of them), which makes the experiment much less convincing. And it is especially critical since this paper heavily relies on empirical justification.

  8. The ablation study is incomplete. For example, to empirically justify the effectiveness of RGE, at least another experiment w.o. RSPAR and RGE and with VGA needs to be provided.

  9. I mentioned this in 4 but reckon it's necessary to have a separate entry for this: As a paper for image registration, there's not any quantitative results on the registration but only on the correspondence. A nice-looking correspondence chart doesn't always guarantee good registration performance. For well tuned SOTA methods (this leads to another issue, see next entry), the correspondence charts usually all look very good, so a further comparison on actual registration quality is needed.

  10. From what I can read from the paper, most of the baselines are not finetuned with the same data or at least on retinal images. This significantly weakens the baselines and the credibility of this paper.

Questions for authors

See weakness section

Reviewer_z3yXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multimodal retinal image registration across CF, FFA, ICGA, and FAF, focusing on the scarcity of accurately registered multimodal training pairs. Its central premise is that effective registration supervision depends more on exact vascular geometric correspondence than on photorealistic cross modal appearance.

The first contribution is Seg2Reg with Retinal Structure Preserving Appearance Randomization, RSPAR. Starting from 733 vessel masks collected from five public segmentation datasets, RSPAR applies structural augmentation and independent fixed and moving deformations while preserving an exact fixed to moving warp. The moving warp includes rotation within ±3°, scale in [0.97, 1.03], and translation within ±2%, while the fixed warp remains near identity. Independent appearance rendering produces deliberately non realistic image pairs without altering their vascular correspondence. A quality gate further removes synthetic pairs with insufficient valid keypoint correspondences, yielding 73,300 training pairs. These exact correspondences supervise vessel aware keypoint detection, feature matching, and correspondence reliability. The SuperPoint encoder and descriptor head remain frozen, while its detector head is adapted, and LightGlue is trained from random initialization.

The second contribution is the Retinal Vascular Geometric Consistency Module, which replaces conventional RANSAC based geometric estimation. Candidate correspondences are represented using coordinates, displacement, matching score, descriptor similarity, detector confidence, and local feature context. Vascular Geometry Aware Attention combines these features with pairwise geometric relations and endpoint confidence to predict correspondence reliability. Reliability Guided Geometric Estimation then filters low reliability matches, enforces spatially balanced correspondence selection, and estimates the transformation using a robust homography followed by support aware quadratic polynomial refinement and bounded local correction.

The method is evaluated zero shot on CFFA, containing 59 CF to FFA pairs, and the private MICRO dataset, containing 1,108 image pairs from 301 eyes. Seg2Reg achieves the best reported performance among the evaluated methods, with RMSE, mAUC, and SR of 2.669, 0.902, and 0.989 on CFFA, and 4.433, 0.665, and 0.787 on MICRO. The ablation study shows that RSPAR provides the largest initial improvement, while RVGC, particularly its geometric estimation component, further improves registration accuracy. Additional experiments evaluate synthetic generation strategies, reliability calibration, learned correspondence filters, cross matcher generalization, RGE components, and computational cost.

Critical strengths

S1. RSPAR provides a large and well-motivated improvement in synthetic supervision. Table 2 shows that adding RSPAR training substantially improves the SuperPoint/LightGlue/RANSAC-H pipeline, increasing mAUC from 0.5878 to 0.8719 and SR from 0.7627 to 0.9831 on CFFA, and from 0.4696 to 0.6298 and 0.6543 to 0.7650 on MICRO. Importantly, Seg2Reg is trained without real benchmark images or landmarks. Table 3 further compares RSPAR with GAN and diffusion based cross modal synthesis under the same SuperPoint, LightGlue, RANSAC-H evaluation pipeline. RSPAR performs better than both modality generation alternatives, supporting the paper's central claim that exact structural correspondence can be more useful for registration supervision than realistic appearance synthesis. This makes RSPAR a substantive contribution rather than a minor modification, although the strength of the generative comparison is limited by the issues discussed in W2.

S2. The experimental reporting is unusually transparent and detailed. Appendix A.2 explicitly reports that, on MICRO, raw RANSAC-H achieves a higher SR than RVGC, 0.8375 versus 0.7744, and explains that conservative reliability filtering can leave some pairs with insufficient correspondences for a valid transformation. This is an important negative result rather than an omitted failure case. The paper also specifies explicit RGE fallbacks, reports Seg2Reg results over three training seeds, uses paired permutation tests with eye-level clustering on MICRO, and provides detailed synthesis and quality-gate hyperparameters. These choices improve the credibility and interpretability of the evaluation. The main RVGC results are also numerically consistent across Tables 2 and 5 up to rounding. One unresolved issue is that the raw RANSAC-H results differ between Tables 2 and 3 and Table 5, which should be clarified rather than assumed to be an error.

Critical weaknesses

W1. The evidence for RVGC is substantially weaker than that for RSPAR, and the current evaluation makes its benefit difficult to interpret.

Table 5 provides the most direct comparison using the same RSPAR trained frontend and candidate correspondences. On MICRO, replacing raw RANSAC H with RVGC improves mAUC from 0.6434 to 0.6568, but reduces SR from 0.8375 to 0.7744. The same SR decrease appears in the cross matcher experiment for SuperGlue, MambaGlue, and GlueStick. Thus, RVGC improves registration accuracy among many evaluated cases, but appears to reduce the probability of obtaining a successful registration on the larger benchmark.

The RMSE reduction does not fully resolve this concern because RMSE is averaged only over acceptable samples, whereas failed and non acceptable cases are counted as failures for SR and AUC. Consequently, a method with lower SR may also change the subset over which RMSE is computed, making RMSE unsuitable as a stand alone measure of overall robustness. This issue is visible in Table 1, where SIFT achieves the second lowest MICRO RMSE despite an SR of only 0.123 and mAUC of 0.096.

The stronger evidence comes from mAUC, which includes failed cases. Here, most of the improvement is attributable to RSPAR. On CFFA, mAUC increases from 0.5878 to 0.8719 with RSPAR and then to 0.9112 with the full model. On MICRO, the corresponding progression is 0.4696 to 0.6298 to 0.6568. Using the stronger raw baseline in Table 5, the additional RVGC gain is only 0.0265 on CFFA and 0.0134 on MICRO. This does not invalidate RVGC, but the empirical support is considerably weaker than for RSPAR.

There is also an important inconsistency that should be clarified before interpreting the SR tradeoff. Table 2 and Table 3 report SR 0.7650 for the RSPAR trained LightGlue plus RANSAC H configuration on MICRO, whereas Table 5 reports 0.8375 for what is described as the same RSPAR trained frontend with raw candidates and RANSAC H. The corresponding CFFA numbers also differ. These values lead to materially different conclusions about whether RVGC improves or reduces success rate. In addition, the statement that RVGC alone shows “limited gains” is misleading on CFFA, where the RSPAR free RVGC configuration reduces both mAUC and SR relative to the baseline.

W2. The experiment supporting the paper's central supervision hypothesis is confounded by unequal training data, while the closest prior methods are not included in the main comparison.

Table 3 is intended to support the claim that exact structural correspondence is more useful than realistic cross modal appearance synthesis. However, the GAN variant is trained on only 600 image pairs, whereas RSPAR provides 73,300 pairs, a difference of more than two orders of magnitude. The number of diffusion generated pairs is not reported, and the diffusion model itself is neither identified nor cited. Therefore, the current experiment cannot isolate the effect of synthesis strategy from the amount of supervision. A stronger test would match the number of pairs and training steps across GAN, diffusion, and RSPAR, and additionally vary the strength of RSPAR appearance randomization while holding geometry fixed.

The main comparison also omits the methods most directly related to the paper's synthetic supervision setting. Section 2 discusses Sindel et al. 2022, which trains retinal feature matching from synthetic multimodal pairs with known homographies, and VesselCycleGAN, yet neither is evaluated in Table 1. Since these methods are closer to the paper's central contribution than several generic matchers, either a controlled comparison or a clear explanation of why they cannot be evaluated under the same protocol would substantially strengthen the novelty claim.

Several baseline reporting details also need clarification. SURF PIIFD RPM is evaluated without a corresponding citation or implementation description. GLAMpoints has no CFFA result and no explanation for the missing entries. The paper also reports competing methods from a single official implementation or checkpoint evaluation, whereas Seg2Reg is reported over three training seeds. This may be reasonable for deterministic pretrained baselines, but the protocol should be documented more clearly.

Finally, reproducibility is limited by the current release plan. MICRO is private and the paper does not describe how its landmarks were annotated, how many landmarks are available per pair, or the original image resolutions relevant to the pixel based thresholds. More importantly, the authors explicitly state that the released random deformation implementation differs from the historical configuration used for the reported checkpoints, while the trained checkpoints themselves are not included. This prevents faithful reproduction of the core training results from the supplied artifacts. The paper should also clarify whether the 73,300 generated Seg2Reg Syn pairs themselves will be released or only the code used to generate them.

Other weaknesses.

  1. Reliability evaluation is limited to synthetic data. Table 4 evaluates (q_{\mathrm{match}}) on 1,000 independently generated RSPAR pairs. Although this is an independent synthetic holdout, it follows the same generation process used to provide (L_q) supervision. No calibration or reliability analysis is reported on real multimodal correspondences. The large ROC AUC improvement is therefore convincing on synthetic data but not yet established in the deployment domain. The relatively small Spearman improvement, 0.3838 versus 0.3651, further suggests that the main benefit is binary reliability discrimination rather than finer error ranking.
  2. No modality specific results are provided. The motivation explicitly includes CF, FFA, ICGA, and FAF, while MICRO contains multiple modalities, diseases, and acquisition systems. However, all MICRO results are aggregated, making it impossible to determine whether the reported gains generalize across modality pairs or are concentrated in particular subsets.
  3. Failure analysis is insufficient. Approximately 21 percent of MICRO pairs do not satisfy the reported success criterion, and Appendix A.2 acknowledges that RVGC can introduce additional failures through conservative filtering. However, the paper provides no systematic breakdown of these failures, and the qualitative analysis contains only one CFFA example.
  4. Several implementation critical details remain underspecified. The manuscript does not provide explicit formulas for the four components of (g_{ij}), the construction of the endpoint bias, or the support aware correction (\Delta(x)). It also does not specify the rejection thresholds used for iterative polynomial fitting or the conditions defining an unreliable local correction. These omissions make independent implementation of RGE difficult.
  5. Several important hyperparameters lack sensitivity analysis. The choices (\tau_q=0.7), the (4\times4) spatial grid, the correspondence cap, and the RVGC training gate thresholds are stated without justification. Appendix A.5 discusses the correspondence cap primarily in terms of latency rather than registration accuracy. The paper should clarify how these parameters were selected and whether evaluation data influenced their choice.
  6. The statistical analysis would benefit from effect size reporting. The paper reports adjusted (p=0.0001) for the mAUC comparisons, but this provides little information beyond statistical significance. Confidence intervals or paired effect sizes would better quantify the practical improvement. In addition, the permutation tests use seed averaged Seg2Reg predictions, so training seed variability is not directly propagated into the hypothesis test.
  7. Minor presentation issue. Figure 4 labels GLAMpoints as “GLAMpoint”, whereas the method is referred to as GLAMpoints elsewhere in the manuscript.
Questions for authors
  1. Table 2, RVGC without RSPAR. RVGC reliability learning requires the exact warp labels generated by RSPAR. In the Table 2 configuration where VGA and RGE are enabled but RSPAR is disabled, what exactly is trained and evaluated? Is the frontend the official LightGlue model, with RVGC trained separately on RSPAR pairs and then applied to its candidates? Also, Section 4.3 describes this configuration as showing “limited gains,” whereas on CFFA it decreases mAUC from 0.5878 to 0.4461 and SR from 0.7627 to 0.5763. Is this description intended?
  2. Inconsistent raw RANSAC H baseline. Table 5 reports MICRO mAUC 0.6434 and SR 0.8375 for raw candidates plus RANSAC H using the RSPAR trained frontend. However, the apparently corresponding RSPAR only configuration in Table 2 and LightGlue RSPAR in Table 3 report mAUC 0.6298 and SR 0.7650. What accounts for this difference, and which result should be used as the controlled baseline when assessing RVGC?
  3. Failure aware evaluation. Since RMSE is computed only over acceptable samples, could the authors additionally report an error measure that includes all pairs, for example by assigning a fixed penalty to failures or reporting an appropriate failure aware statistic? In particular, how much of the MICRO RMSE reduction from 6.3415 to 4.3125 remains under such an evaluation, and how many additional pairs fail to produce a valid transformation after RVGC reliability filtering?
  4. Controlled comparison in Table 3. Which diffusion model is used for LightGlue Diffusion, and how many generated training pairs does it contain? Could the GAN, diffusion, and RSPAR variants be compared using matched numbers of training pairs and optimization steps? It would also be useful to vary the strength of RSPAR appearance randomization while keeping the geometric supervision fixed, since this would more directly test the claim that strong appearance randomization is preferable to realistic appearance synthesis.
  5. MICRO ground truth and evaluation scale. How were the MICRO landmark correspondences established, including the annotation procedure, number of landmarks per pair, and any assessment of annotation consistency? Please also report the original image resolutions for CFFA and MICRO. Since the 20 and 50 pixel success thresholds and the 1 to 25 pixel AUC thresholds are evaluated in original image coordinates, these details are important for interpreting the reported errors across datasets.
  6. Hyperparameter selection and sensitivity. How were (\tau_q=0.7), the (4\times4) spatial grid, and (K=256) selected, and were any evaluation images or landmarks used during this process? In particular, could the authors report mAUC and SR sensitivity to (\tau_q)? This threshold directly controls the number of retained correspondences and therefore appears closely related to the success rate tradeoff observed on MICRO.
  7. Release and reproducibility. Will the 73,300 Seg2Reg Syn pairs and the trained checkpoints be released? The Reproducibility Statement notes that the archived random deformation implementation differs from the historical configuration used to train the reported checkpoints. Can the released implementation be aligned with the configuration underlying the reported results so that the main training experiments can be reproduced faithfully?
  8. Baseline configuration. Please clarify how SuperRetina was evaluated under the common 1024 by 1024 protocol, including whether its official pretrained model was used directly or whether any retraining or adaptation was performed. Please also explain why GLAMpoints has no CFFA result and provide a citation and implementation details for SURF PIIFD RPM.
Reviewer_4VzgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Structure-Preserving Appearance Randomization for Multimodal Retinal Image Registration

Critical strengths

This paper proposes a Seg2Reg learning framework that employs Retinal Structure-Preserving Appearance Randomization (RSPAR) to construct a synthetic registration dataset from retinal vessel segmentation data.

Critical weaknesses

The compared methods are relatively outdated, and the experimental evaluation lacks comparisons with more recent state-of-the-art registration methods. This limits the fairness and comprehensiveness of the performance evaluation. The authors are encouraged to include more competitive and recent baselines to better demonstrate the superiority of the proposed method.

The practical significance of multimodal retinal image registration is insufficiently discussed. Although the authors briefly mention its potential clinical applications, the manuscript does not clearly explain how accurate registration benefits downstream tasks, such as disease diagnosis or treatment planning. A more detailed discussion, preferably supported by representative application examples, would strengthen the practical motivation and clinical relevance of this work.

The robustness of the proposed framework to imperfect vessel segmentation annotations remains unclear. Since RSPAR relies on retinal vessel masks to generate synthetic training pairs, segmentation errors may affect the quality of the generated data and subsequent registration performance. The authors are encouraged to investigate the sensitivity of Seg2Reg to vessel segmentation quality, for example, by using masks produced by different vessel segmentation models or introducing controlled segmentation errors.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.