Mind the Margin: When Cross-Modal Geometry Misleads Retrieval

Post-hoc transformations of pretrained cross-modal embeddings, such as compressions and alignment maps, change retrieval in ways the standard measures of cross-modal structure do not register. Across $98$ encoder pairs, three datasets, four modalities and $16$ transformations, alignment maps close the modality gap by $63$–$91\%$ on MSCOCO while removing $18$–$73\%$ of Recall@$1$, and CKA and canonical correlation stay exactly unchanged under an orthogonal map that removes $40\%$ of it. We explain the loss through the comparisons that decide retrieval. A transformation perturbs the similarity difference between two candidates, and we read off the ambient margin below which orderings become unreliable, the order-preservation threshold. Under a location–scale model, the expected loss depends on the transformation only through this scale, its severity, and on the target only through its top-two similarity gaps, its exposure. We show supporting evidence for the account. The threshold is a property of the transformation, its median stable to within $2\%$ from $500$ to $5,000$ gallery items, and estimated without labels on reference encoder pairs it orders the losses of the compressions at $\rho =0.72$ to $0.90$ on three datasets. Read against it, a target's gaps order targets under a fixed compression at a mean $\rho$ of $0.59$ on MSCOCO and $0.53$ on MSR-VTT, and order $29$ encoder pairs by their mean loss at $\rho =0.76$, where ambient accuracy and separability do not. Together they forecast a transformation's cost before it is applied, from an unlabelled retrieval log, for settings in which the target's embeddings are out of reach.