SAFR: Foundation-Error-Aware Ranking and Sparse Affinity Residualization for Multi-modal Object Re-Identification

Multi-modal object Re-Identification (ReID) combines RGB, near-infrared, and thermal-infrared observations to match identities across cameras or scenes. A frozen visual encoder can rank a look-alike distractor above the true match when the latter is seen from a different viewpoint. We propose Sparse Foundation-Affinity Residualization (SAFR), which uses frozen-model reversals to shape training margins and frozen affinities to correct retrieval scores. During training, identity labels reveal such reversals, and foundation-error-aware ranking increases the task margin in proportion to their magnitude while retaining standard identity supervision. After training, sparse affinity correction estimates, on different-identity training pairs, the part of the task score associated with global and local frozen affinities. Held-out training identities determine which contributions to subtract, yielding a fixed retrieval score. Experiments obtain 83.7%, 89.6%, and 71.5% mAP on RGBNT201, RGBNT100, and MSVR310, respectively. Component and mechanism analyses further show how representation learning and score correction contribute to the final retrieval performance.