acceptodds
Under review as a conference paper at ICLR 2027

SAFR: Foundation-Error-Aware Ranking and Sparse Affinity Residualization for Multi-modal Object Re-Identification

Abstract

Multi-modal object Re-Identification (ReID) combines RGB, near-infrared, and thermal-infrared observations to match identities across cameras or scenes. A frozen visual encoder can rank a look-alike distractor above the true match when the latter is seen from a different viewpoint. We propose Sparse Foundation-Affinity Residualization (SAFR), which uses frozen-model reversals to shape training margins and frozen affinities to correct retrieval scores. During training, identity labels reveal such reversals, and foundation-error-aware ranking increases the task margin in proportion to their magnitude while retaining standard identity supervision. After training, sparse affinity correction estimates, on different-identity training pairs, the part of the task score associated with global and local frozen affinities. Held-out training identities determine which contributions to subtract, yielding a fixed retrieval score. Experiments obtain 83.7%, 89.6%, and 71.5% mAP on RGBNT201, RGBNT100, and MSVR310, respectively. Component and mechanism analyses further show how representation learning and score correction contribute to the final retrieval performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-modal Object Re-Identification, Foundation Models, Similarity Calibration, Visual Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R986OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the problem that pretrained visual models tend to rank visually similar distractors ahead of true matches, the authors propose the SAFR framework. During training, the SAFR framework introduces Foundation-Error-Aware Ranking, which uses ranking errors from a frozen foundation model to dynamically adjust the margin constraints on hard samples, thereby improving the model's ability to distinguish identities. During retrieval, it employs Sparse Foundation-Affinity Correction, which combines global and local feature affinities to correct the final matching scores. Experiments on RGBNT201, RGBNT100, and MSVR310 show that SAFR outperforms the comparison methods listed in the tables.

Critical strengths
  1. Section 3.1 proposes a method that uses incorrect rankings of positive and negative samples from a frozen foundation model to adjust the training margins, rather than directly learning the foundation model's similarity relations. I think using these ranking errors to improve the model's ability to distinguish identities is a good idea.
  2. Table 4 in Section 4.4 compares different retrieval score correction methods. These detailed comparisons help better understand the actual impact of affinity correction on retrieval results and demonstrate the authors' solid experimental work.
Critical weaknesses
  1. The main ablation experiments in Section 4.4 are all conducted on RGBNT201, without corresponding analyses on the two vehicle Re-ID datasets, RGBNT100 and MSVR310.
  2. In Table 5 of Section 4.4, using only global affinity already achieves 83.5% mAP, while adding local affinity improves it only to 83.7%, with no change in Rank-1. This result makes me skeptical about the actual contribution of local features.
Questions for authors
  1. Regarding the foundation model's ranking reversal magnitude in Eq. (7), does the performance improvement actually come from the error information provided by the foundation model, or simply from imposing stronger ranking constraints on hard samples?
  2. The main ablation studies in Section 4.4 are limited to RGBNT201. Do the two proposed components provide similar improvements on RGBNT100 and MSVR310?
Reviewer_iyrVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SAFR, which leverages frozen foundation-model ranking errors and affinities to improve multimodal object ReID through training-margin adjustment and retrieval-score correction.

Critical strengths

The paper provides comprehensive experiments and demonstrates good awareness of comparison fairness.

Critical weaknesses
  1. The motivation is not sufficiently convincing, as the proposed method does not explicitly address the challenges of multimodal object ReID. DINOv3 is pretrained on RGB images, so directly applying it to NIR/TIR data introduces potential out-of-distribution issues. The paper treats ranking errors as informative signals, but these may reflect the foundation model’s failure on non-RGB inputs rather than the claimed semantic discrepancies between pretrained relationships and identity labels
  2. The proposed post-processing pipeline is overly complex and engineering-heavy. Although the Related Work discusses k-reciprocal and metric-fusion re-ranking, the paper does not directly compare SAFR with these lightweight alternatives, making it difficult to assess whether the complex correction pipeline offers substantial benefits over simpler re-ranking methods.
Questions for authors

My main concern is the validity of the underlying motivation. The intrinsic connection between the proposed method and multimodal learning remains unclear.

Reviewer_mLKZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework named SAFR for multi-modal object re-identification that leverages ranking errors and pairwise affinities from a frozen visual foundation model. Foundation-Error-Aware Ranking adjusts training margins based on incorrect positive-negative orderings from the encoder, while Sparse Foundation-Affinity Residualization subtracts reference contributions from retrieval scores using coefficients estimated from cross-validation. Experiments on three benchmark datasets demonstrate strong empirical performance. However, despite its interesting perspective, the individual components and their advantages over simpler ranking or score-calibration baselines require deeper justification.

Critical strengths
  1. Instead of forcing standard knowledge distillation or feature alignment, SAFR exploits the erroneous similarity rankings of a frozen foundation model to guide identity discrimination. Besides, combining error-conditioned margin adjustments with negative-pair affinity residualization, alongside identity-disjoint cross-validation, provides a reasonable way to avoid overfitting to test identities.

  2. The empirical evaluations across three challenging benchmarks with strong baselines illustrate the effectiveness of the proposed model.

Critical weaknesses
  1. The foundation-error-aware ranking contributes very little, as shown in Figure 3, dropping the error-conditioned margin has minimal impact. Stronger comparisons with standard hard-example mining, ranking losses, and supervised score fusion are required.
  2. The correction coefficients are fitted via statistical associations among different-identity pairs, which does not guarantee that the actual foundation-model errors are being targeted.
Questions for authors
  1. What is the actual independent contribution of the error-aware ranking? Can you compare the error-conditioned margin directly against standard smooth ranking, hard-example weighting, and alternative margin adjustments under the exact same score-correction setup?
  2. Why is affinity residualization better than traditional score calibration or post-hoc re-ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.