acceptodds
Under review as a conference paper at ICLR 2027

Sparse-Landmark Non-Rigid Alignment of Spectral Point Sets for Cross-Modal Retrieval

Abstract

Cross-modal retrieval typically learns shared representations from abundant paired data. We study a complementary regime in which unimodal representations are independently constructed and kept fixed, while reliable cross-modal correspondences are scarce. The resulting challenge is therefore not representation learning, but sparse registration between two latent spaces. Our key observation is that independently learned spectral spaces can retain non-random semantic neighborhood agreement despite occupying different coordinate frames. This motivates modeling their mismatch as a dominant global coordinate discrepancy followed by residual, position-dependent non-rigid deformation. We accordingly formulate sparse cross-modal alignment as a global-to-local geometric registration problem and propose Sparse-Landmark Non-Rigid Alignment (SLNA). Using verified pairs as sparse landmarks, SLNA first recovers a global distance-preserving coordinate relation and then allocates nonlinear capacity only to the remaining residuals through regularized corrections in complementary coordinate and angular geometries. Experiments on Flickr30k and MSCOCO show that SLNA consistently improves over rigid Procrustes alignment and also outperforms a direct RBF mapping under matched model-selection settings. Structural diagnostics reveal statistically significant cross-modal neighborhood agreement, while controlled-deformation experiments show that residual modeling recovers local distortions beyond a single global transformation. These results suggest that sparse cross-modal alignment benefits from treating global coordinate recovery and local deformation modeling as distinct estimation problems, rather than learning a single flexible mapping from sparse pairs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
cross-modal retrieval、sparse-pair supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rj8BOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sparse cross-modal retrieval from a geometric alignment perspective. The main idea is to treat independently learned image and text representations as two latent spaces that need to be aligned. Based on this, the authors propose SLNA, which first uses sparse verified pairs to estimate a global Procrustes alignment, and then models the remaining local mismatch with RBF-based non-rigid corrections. An optional soft refinement further makes use of unverified candidates. Experiments on Flickr30k and MSCOCO show clear improvements over rigid alignment and modest but consistent gains over direct nonlinear mapping, together with additional analyses on landmark efficiency, structural consistency, and robustness.

Critical strengths
  1. I find the geometric formulation of sparse cross-modal alignment quite intuitive, especially the separation between global coordinate mismatch and local non-rigid deformation.
  2. SLNA combines Procrustes-based global alignment with regularized local residual modeling, making the overall pipeline easy to follow and implement.
  3. I also appreciate that the evaluation goes beyond retrieval accuracy and includes analyses of structural consistency, landmark efficiency, geometry preservation, robustness, and synthetic deformation recovery, which makes the geometric interpretation more convincing.
  4. The method works with fixed unimodal encoders and only a small number of verified pairs, which is a meaningful setting when paired cross-modal data is limited.
Critical weaknesses
  1. The motivation for using both the coordinate and angular branches is reasonable, but their complementary roles could be explained more clearly. In particular, the different fusion weights selected for Flickr30k and MSCOCO are interesting and deserve a short discussion.
  2. SLNA-soft helps on some metrics but leads to small drops on others. It would be useful to hear the authors’ view on when the soft refinement is likely to be beneficial, and when the base SLNA model may be a safer choice.
  3. The main evaluation is based on independent 400-sample test sets, and an identity-matched comparison with SUE is not available. It would be useful to clarify how representative these subsets are of the full datasets.
  4. Since both the global alignment and the local residual model rely on sparse landmarks, I wonder how much performance depends on landmark coverage rather than just the number of landmarks. Some discussion of the role of spatial or semantic coverage in the spectral space would be helpful.
  5. The synthetic deformation experiment shows that the residual model can recover non-rigid distortions. However, the connection between the synthetic deformation and the residual mismatch observed on Flickr30k and MSCOCO could be made clearer.
  6. The paper uses d=30d=30 for all spectral representations, but the effect of this choice is not discussed in detail. A brief explanation of why this dimensionality is appropriate, and whether the method is expected to be sensitive to moderate changes in dd, would improve the technical clarity.
Questions for authors
  1. Could the authors clarify how sensitive SLNA is to imperfect global alignment? Since the local residual field is fitted after the Procrustes stage, it would be useful to understand whether moderate errors in the estimated global transform can be effectively absorbed by the local RBF correction, or whether they noticeably degrade performance.
  2. Could the authors provide more intuition on the dataset-dependent fusion between the coordinate and angular branches? In particular, the selected βraw\beta_{raw} differs substantially between Flickr30k and MSCOCO. Is this mainly due to differences in the geometry of the learned spectral spaces, or simply an outcome of cross-validation?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

We sincerely appreciate the reviewer's thoughtful and constructive feedback, which has helped us further clarify our methodology, strengthen our experimental analysis, and improve the overall quality of our manuscript. We hope that our detailed responses and additional experimental evidence have adequately addressed your concerns, and we would be grateful to know whether any questions remain.

We also welcome any further questions or suggestions you may have and will be happy to provide additional clarifications and engage in further discussion.

Best regards,

SLNA Authors

Reviewer_VR9UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents sparse cross-modal representation alignment as the point cloud registration. It uses the global procrustes as the stable prior, then establish the local non-rigid residuals model using regularized RBF.

Critical strengths
  1. It is clear to define the problem as multi-scale geometric mismatch problem. The paper clearly distinguishes representation learning and frozen-space alignment.
  2. The alignment method from the global to the local is a reasonable approach.
  3. The presentation of the figure is great.
Critical weaknesses
  1. The contribution of the SLNA is not fully, the increase is relatively limited. Also, the direct RBF method lacks the global mapping terms, it is not the most fair comparison.
  2. The part of relative work should contain the reference of non-rigid registration. This part is too short.
  3. There is no separate ablations of the raw-coordinate branch, the angular branch, the chord/tangent target and the fusion.
Questions for authors
  1. Due to the limited improvement of direct RBF. Why it supports the effectiveness of the proposed global–local decomposition?
  2. Can the method be full ablations?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

We are sincerely grateful to the reviewer for your thoughtful suggestions and insightful feedback, which have encouraged us to further refine our methodology and strengthen the supporting experimental evidence. We hope our revisions and explanations have satisfactorily resolved your concerns, and we would be grateful to hear whether any questions remain.

We also welcome any further questions you may have and will actively respond to them.

Best regards,

SLNA Authors

Reviewer_o9JcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sparse-Landmark Non-Rigid Alignment (SLNA), a framework for cross-modal retrieval with limited verified image-text correspondences. The key idea is to decompose cross-modal alignment into two stages: global rigid alignment using Orthogonal Procrustes, followed by local non-rigid correction using regularized radial basis functions (RBFs). The method further combines coordinate-based and angular residual modeling, with an optional posterior-based refinement (SLNA-soft) leveraging unverified samples. Experiments on Flickr30k and MSCOCO demonstrate improvements over rigid alignment and other baselines.

Critical strengths

The global-to-local decomposition provides an intuitive perspective on sparse cross-modal alignment. Moreover, the framework is computationally manageable and does not require updating the pretrained encoders, making it potentially useful in practical low-supervision scenarios.

Critical weaknesses

Although SLNA substantially outperforms Orthogonal Procrustes, its improvements over the Direct RBF baseline are relatively modest. Since the primary methodological contribution is the explicit global-to-local decomposition, stronger evidence is needed to establish its consistent advantage over a carefully regularized direct nonlinear mapping.

Questions for authors
  1. Could the authors provide a more fine-grained ablation separating the contributions of coordinate-space residual correction, angular residual correction, and their fusion?
  2. Could the authors provide an identity-matched comparison with SUE and other sparse-pair alignment baselines, using identical feature representations, landmark selections, and evaluation sets?
  3. How does SLNA perform under standard large-scale retrieval protocols?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

We sincerely thank the reviewer for your valuable and constructive comments, which have helped us strengthen our experimental evaluation and improve the overall quality of our manuscript. We hope that our detailed responses and additional experiments have adequately addressed your concerns. We would greatly appreciate any further feedback and welcome continued discussion.

We also welcome any further questions you may have and will actively respond to them.

Best regards,

SLNA Authors

Thank you for your reply. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.