- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper connects the SigLIP loss to an estimator of pointwise mutual information and applies this finding to derive a novel approach for multimodal medical image registration (or rather a plug-in loss usable with existing frameworks). The approach relies on synthetic data following the domain randomization paradigm. It is evaluated on 4 multimodal multi-organ datasets including US/CT/MRI images, on top of 3 registration frameworks. It is then applied and compared end-to-end vs. full registration frameworks, showing promising results.
Critical strengths
The paper presents a rigorous analysis of the SigLIP loss as a pointwise mutual information estimator biased by a class prior directly related to batch size. I found this analysis insightful and clearly presented. Although I am not sure it is entirely novel, I could not clearly tie this analysis to a specific previous paper.
Secondly, the paper connects those insights to a practical registration framework and shows promising results. The authors also make a sensible use of convolutional architectures with known receptive field to train the PMI estimator at patch-level and apply it on full volumes in convolutional manner, which is a sound and practical approach.
Finally, the paper is clear, well structured and easy to follow.
Critical weaknesses
The viability of the resulting domain randomization + PMI estimator registration strategy is not fully established by the paper, but in my opinion this is acceptable given its theoretical contribution.
A weakness of the approach is its reliance on multimodal data with perfect alignment (i.e., known ground-truth displacement). This virtually constrains the approach to in fact follow this domain randomization strategy with synthetic data, and limits its scope.
However at this stage, suspected weaknesses are not substantive and I will move them to open questions instead.
Questions for authors
-
One of my concerns is that while the SigLIP based PMI estimator is theoretically well-grounded, it would be in practice difficult to train due to the constraint on batch size. Could authors make more precise the typical batch size at which they manage to work on 3D patches? If you extract too few random patches from typically large volumes, minimizing the SigLIP loss (i.e. identifying positives vs. negatives) might be trivial due to the large diversity of appearance of patches (making it likely that all sampled pairs heavily differ from pair to pair). I would assume that if the patch positives vs. negatives can be trivially identified, the approximation quality of the PMI degrades? Can you also comment on the choice to use non-overlapping patches vs. constraining the maximum overlap?
-
Can authors comment on the ellipsoidal masking step and how it affects the reliability of the loss "matching" anatomical regions cross-modality? The masking seems to introduce irrealistic black regions and boundaries in patches, which would deviate from the data observed in inference? To what extent is the trained model affected by this or its performance upper-bounded by this?
-
Can the authors comment on the trade-offs in choosing the receptive field for the estimator (here R9 or R17, but many other values could have been considered a priori). Does the performance change gracefully with higher receptive field values, or does it become impractical to train because of patch size or shortcuts? What about values between 9 and 17?
3 comments
Author Response (Part 2)
"Can you also comment on the choice to use non-overlapping patches vs. constraining the maximum overlap?"
Patches that share voxels are dependent and cannot be samples from the product of marginals. Labeling them as negatives would thus hurt the estimator. Labeling them as positives would make the score saturate before alignment is reached. A maximum overlap would trade a small bias for harder negatives, and our augmentations already allow occasional small overlaps, which we accept as minor training noise (Appendix B.1). Graded supervision by overlap would go beyond PMI estimation, and we leave it for future work.
"Can authors comment on the ellipsoidal masking step [...]? The masking seems to introduce irrealistic black regions and boundaries in patches, which would deviate from the data observed in inference? To what extent is the trained model affected by this or its performance upper-bounded by this?"
The masks mimic field-of-view borders, which do occur at inference, for example at the edge of the ultrasound cone in RESECT and TRUSTED, or with partial coverage in CT and MRI. Because each patch receives an independent, smoothly deformed ellipsoidal mask, training covers pairs in which one patch shows anatomy that the other only partly shows, as at a field-of-view border during registration. Masks are applied with probability 0.7, leaving 30% of training pairs unmasked (Appendix B.1).
The new ablation retrains all three variants without masking (Appendix C.6, Table 7). Masking never significantly hurts, and its benefit grows with the receptive field: at R9 it changes results by at most 0.09 Dice points and 0.16 mm, while at R33 it significantly improves Abdomen by 11.06 Dice points, RESECT by 4.54 mm, and TRUSTED by 0.39 Dice points. On LuMIR, where both modalities cover the same anatomy, it changes R9 and R33 by at most 0.03 Dice points and significantly improves R17 by 0.26. We therefore see no sign that the masked regions limit performance on unmasked data.
"Can the authors comment on the trade-offs in choosing the receptive field for the estimator [...]? Does the performance change gracefully with higher receptive field values, or does it become impractical to train because of patch size or shortcuts? What about values between 9 and 17?"
The receptive field trades shared information against local precision. A larger window captures more information shared between modalities, consistent with the learned PMI range widening with R: its upper end grows from 6.1 nats at R9 to 9.6 and 10.6 nats at R17 and R33 (Table 3). This context helps under large misalignment, but each estimate then averages over a larger region and localizes deformations less precisely. Small windows also make negatives harder and reduce the gap between synthetic and real images (Appendix B.2). The new SigMI-R33 results in Table 1 reflect this trade-off. The best SigMI variant is R9 in 6 of the 12 settings, including all three on LuMIR, R17 in 3, and R33 in 3: on Abdomen, which has the largest initial misalignment (32.65% Dice before registration), with MultiresSVF and ConvexAdam, and on RESECT with ConvexAdam. R33 remains practical to train (2.1M parameters, Table 3), but it is more prone to shortcuts, which is where the sigmoid objective and masking matter most (Tables 2 and 7, discussed above).
Values between 9 and 17 would require different kernel sizes per stage: with the same kernels in all four stages, (Appendix B.2), which gives 9, 17, 25, and 33. Given this trade-off, and since MultiresSVF and FireANTs SyN are multi-resolution, a natural extension is to use larger receptive fields at coarse levels and smaller ones at fine levels.
Thank you again for your questions, which improved the paper. If our answers address them, we would be grateful if you would consider raising your score, and we would be glad to discuss any further questions.
Many thanks to the authors for taking the time to answer my comments and questions. I am satisfied with these responses and updates, which address the points I raised. On the other hand, my initial review underestimated a bit the overlap of the theoretical part with existing papers, as pointed out by other reviews. All put together, I will maintain my initial rating as my assessment is that the paper makes contributions of interest to the community in the context of multimodal image registration.
Author Response (Part 1)
Thank you for your positive and detailed review, and for finding our analysis of SigLIP rigorous, insightful, and clearly presented, and our fully convolutional design sound and practical. Your questions led to three key additions in the revised manuscript: SigMI-R33 in Table 1, a field-of-view masking ablation (Appendix C.6), and a batch-size study against InfoNCE (Section 4.3, Figure 5). Revised text is in blue.
Thank you. The revision strengthens this evidence. Table 1 now reports paired significance tests: of the 72 baseline–setting comparisons, 51 are significantly worse than SigMI and 4 significantly better. While the strongest baseline changes with the dataset, SigMI-R9 alone has the best average rank over the 12 settings (1.8, against 2.8 for squared LNCC and 2.7 for MIND-SSC). Against complete registration methods (Section 4.4), SigMI with MultiresSVF has the second-best median on both LuMIR and RESECT, and the best mean TRE on RESECT, without any real training image.
We agree that sampling the joint distribution requires perfectly aligned pairs, which is why we train on synthetic data only. We see this less as a limit on scope than as what makes the approach general. Because SigMI scores local patches, the synthetic data only need to be realistic at the patch scale, not as whole volumes, unlike networks trained on full synthetic volumes such as SynthMorph (Appendix B.2). In particular, five brain label maps suffice for an estimator that transfers to real MRI, CT, and ultrasound and to abdominal anatomy, without seeing any real image during training (Table 1). Where aligned real pairs exist, they can be added to the training data without changing the method.
The batch size is not a constraint for SigMI. In theory, the prior correction makes the optimal score PMI for any B (Theorem 2). In practice, memory is not the limit either: a batch of 16,384 R17 patch pairs fits on a single 80 GB H100. We chose B=32 patch pairs, with gradients accumulated over four batches (Appendix B.2), for training speed: at this batch size, SigMI-R9 and SigMI-R17 each train in under 8 hours on one H100, whereas larger batches are slowed down by loading more image pairs. The new Section 4.3 varies B from 4 to 64 for SigMI-R17 (Figure 5). Going down to B=4, its TRE on RESECT and Dice on TRUSTED barely change and its LuMIR Dice drops by less than one point; going up to B=64 changes these three by at most 0.17 Dice points and 0.09 mm. Abdomen, with only 4 test pairs, varies more across batch sizes.
Yes, and this concern shaped our design: trivially separable joint and product-of-marginals samples give strong classifiers but poor density-ratio estimates (Rhodes et al., 2020). We make negatives hard in three ways (Section 3.5, Appendix B.1). Each batch draws spatially connected groups of patches from a few image pairs (at most 16 per image), which makes many negatives neighboring, visually similar patches rather than distant anatomy. Small receptive fields keep patches geometrically simple. Field-of-view masking removes the silhouette as a shortcut.
SigMI's advantage over InfoNCE grows with the receptive field: significant wins are tied 4–4 at R9, and 8–0 and 12–0 for SigMI at R17 and R33 (Section 4.3, Table 2). We attribute this to easy negatives: larger patches are easier to tell apart, and easy negatives hurt InfoNCE, which scores each pair relative to its negatives, more than SigMI, which judges each pair independently. Masking likewise helps more as patches grow (Table 7), since larger patches more often fall on a field-of-view border.
(continued below)