acceptodds
Under review as a conference paper at ICLR 2027

Beyond Ranked Gallery: Conformal Identity Set for Text-Based Person Retrieval

Abstract

Text-based person retrieval enables users to search for individuals in an image gallery using natural-language descriptions. Existing methods primarily focus on cross-modal matching and ranking accuracy, whereas practical retrieval also requires deciding how many candidate identities to return. A fixed cutoff may exclude the target for ambiguous descriptions or return unnecessary candidates for distinctive ones. Conformal prediction offers a principled framework for constructing such sets by calibrating candidate-level scores. A natural approach is to derive these scores from softmax-normalized retrieval similarities. However, this normalization introduces a temperature dependence that can alter set sizes and coverage evenness without changing the retrieval ranking. To understand this dependence, we analyze the low-temperature behavior of representative softmax-based conformal scores. We prove that, after order-preserving transformations, these scores converge to a common similarity-margin score as temperature approaches zero. Motivated by this shared limit, we introduce a temperature-free method for constructing Conformal Identity Sets through direct margin calibration. Specifically, our method retains identities within a common calibrated similarity gap of the top-ranked identity, allowing set sizes to adapt to each query's similarity profile. The resulting sets provide finite-sample marginal coverage at a user-specified level under exchangeability, without softmax, temperature tuning, or retraining. Experiments with five frozen retrievers across three benchmarks demonstrate empirical coverage close to the target and a favorable trade-off between set size and coverage evenness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text based Person Retrieval, Conformal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y2AUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work moves beyond fixed Rank-KK retrieval and asks how many identities should be returned for each query. The authors connect standard conformal scores to a similarity-margin formulation and use this margin to build query-adaptive identity sets without retraining the retriever. Experiments on several TPR models and datasets evaluate the trade-off between coverage and set size.

Critical strengths

1. Clear presentation.

The paper is reasonably well organized and easy to follow. The progression from the practical motivation, to the conformal analysis, and then to the margin-based method is presented in a relatively clear way.

2. Interesting motivation.

The paper looks beyond standard Rank-KK evaluation and considers the practical question of how many candidate identities should be returned for different queries. This is a meaningful direction that is less explored in existing TPR work.

3. Reasonable theoretical analysis.

The paper provides a clear analysis of the temperature dependence of LAC and APS and derives their common low-temperature connection to the similarity margin. The theoretical development is generally coherent and helps motivate the final construction.

4. Comprehensive experiments. The evaluation covers multiple frozen retrievers and three TPR benchmarks, and reports several metrics related to coverage and set efficiency. The paper also includes analyses of temperature behavior, query adaptivity, and different retrieval difficulty groups.

Critical weaknesses

1. Mismatch Between the Query-Ambiguity Motivation and the Actual Adaptation Mechanism

The paper motivates adaptive identity sets from the perspective of query ambiguity, arguing that ambiguous queries should return larger sets while distinctive queries should return smaller ones. However, the proposed method never explicitly estimates query ambiguity. Instead, the set size is determined entirely by the similarity profile produced by the frozen retriever, through the margin between each identity and the top-ranked identity. Therefore, the method is more accurately adapting to retriever uncertainty or score crowding rather than to query ambiguity itself. These two factors are not necessarily equivalent, and the current experiments do not establish a clear connection between them.

2. Limited Methodological Novelty

The final method is relatively straightforward: it calibrates the similarity gap between the target identity and the top-ranked identity on a held-out set, and then returns all identities within the calibrated margin at inference time. While the theoretical connection between LAC, APS, and the similarity margin is interesting, the resulting retrieval mechanism itself is essentially a calibrated thresholding rule over top-1 similarity gaps. The methodological contribution therefore appears limited relative to the development of this task.

3. Strong Dependence on the Identity-Level Max Aggregation

The identity similarity is defined as the maximum similarity over all gallery images belonging to that identity. This seems to discard the remaining observations of the same identity and makes the identity score potentially sensitive to a single spuriously high-matching image.

4. Strong Assumption of Known Gallery Identity Labels

The proposed formulation requires the gallery to be partitioned into identity groups in advance, since identity-level similarity is computed over all images belonging to each known identity. This assumption may be restrictive in practical person retrieval scenarios, where gallery identity labels are typically unavailable.

5. Unclear Practical Benefit of Adaptive Set Sizes

The practical advantage of returning query-dependent set sizes is not sufficiently justified. In many retrieval scenarios, the user has a fixed inspection budget, such as examining only the top-5 or top-10 candidates. Under such a setting, returning a larger set for difficult queries does not necessarily provide additional value: if the target falls outside the inspection budget, the retrieval is still effectively unsuccessful. Therefore, it is unclear under what practical setting the proposed adaptive identity set is preferable to a fixed top-KK retrieval protocol.

6. Gap Between the Low-Temperature Theory and Finite-Temperature Practice

The theoretical motivation for direct margin calibration relies on the limit T→0T \rightarrow 0, where transformed LAC and APS scores converge to the similarity margin. However, the practical LAC and APS baselines operate at finite temperatures selected by calibration. It is therefore unclear whether the fitted operating temperatures actually lie in a regime where the margin provides a close approximation.

Questions for authors

Please refer to the Weaknesses section for the main concerns. I have a few questions that I hope the authors can clarify:

  1. The paper motivates adaptive identity sets in terms of query ambiguity, while the proposed method adapts only to the retriever’s similarity profile. Could the authors clarify why similarity-margin structure should be considered a reliable proxy for query ambiguity?
  2. The method assumes that gallery images are already grouped by identity and defines each identity using only its maximum image similarity. How realistic is this assumption in deployment, and how sensitive are the results to alternative identity-level aggregation strategies?
  3. The theoretical motivation relies on the T→0T\rightarrow0 limit, whereas LAC and APS are used at finite, calibrated temperatures in practice. Could the authors report how close the proposed Margin sets are to LAC/APS specifically at their selected operating temperatures?
Reviewer_Gh16OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors explore an interesting perspective by combining text-based person retrieval with Conformal Prediction. They identify the limitations of existing LAC and APS methods, whose returned candidate sets can be affected by the softmax temperature. To address this issue, they analyze the low-temperature behavior of LAC and APS and propose a temperature-free conformal calibration method based on the similarity margin. The proposed method enables different queries to return adaptive numbers of candidate identities while maintaining the desired marginal coverage guarantee. Experiments on multiple retrievers and benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper presents an interesting problem formulation by introducing conformal prediction into text-based person retrieval. Instead of using a fixed Top-K cutoff, it considers query-adaptive candidate sets with a marginal coverage guarantee, providing a different perspective from conventional ranking-based retrieval.
  2. The paper provides both theoretical analysis and empirical validation. The authors analyze the temperature dependence of LAC and APS, derive their low-temperature connection to the similarity margin, and verify the proposed method across multiple retrievers and benchmarks.
Critical weaknesses
  1. The practical motivation is somewhat weak. The paper mainly focuses on adaptively deciding how many candidates to return, but it does not improve the underlying retrieval ranking itself. In this sense, the work feels somewhat like applying a conformal prediction framework on top of an existing ReID/TBPR system rather than addressing a retrieval-specific limitation. In practice, person retrieval systems already provide a ranked list, so it is still unclear how much practical benefit an adaptive cutoff can bring. The paper shows smaller candidate sets under similar coverage, but the real-world value of this improvement is not sufficiently demonstrated.
  2. Although Margin often reduces the mean set size, it is not uniformly better than all alternatives. For example, the refinement baseline is slightly better in some cases, and LAC can also produce smaller sets, although with worse coverage balance. This makes the advantage of Margin look more like a trade-off rather than a clear improvement.
Questions for authors

My main concern is still about the motivation of this work. The proposed method adaptively determines how many candidates should be returned, but it does not improve the underlying retrieval ranking itself. Could the authors further clarify why adaptive candidate-set size is important in practical text-based person retrieval systems? In particular, compared with simply providing a ranked gallery or using a fixed Top-K list, what concrete practical benefit does the proposed conformal set bring to real users or downstream systems?

1 comment
Area_Chair_KhkiOpenReview ↗

Authors' rebuttal has submitted

Dear reviewer, this is a kind reminder that the authors have posted their responses to your reviews. Please review their rebuttal, engage in any necessary follow-up discussion with the authors, and confirm whether your concerns have been addressed by updating your review/scores. Thank you for you colaboration.

Reviewer_5aa6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a temperature-free conformal prediction method, termed Conformal Identity Sets, for text-based person retrieval. It analyzes the temperature dependence of existing softmax-based conformal scores and theoretically shows that transformed LAC and APS scores converge to a common similarity-margin score at low temperatures. Based on this finding, the method directly calibrates similarity margins to generate query-adaptive candidate sets with finite-sample marginal coverage under exchangeability. Experiments on three benchmarks and five frozen retrievers evaluate its efficiency and coverage behavior

Critical strengths
  1. Provides theoretical insights into the temperature dependence of softmax-based conformal scores and their shared low-temperature limit
  2. Introduces a simple, training-free method that enables query-adaptive candidate selection without temperature tuning
Critical weaknesses
  1. The coverage guarantee relies on exchangeable query episodes, but the experimental protocol partitions identities and treats all captions as separate episodes. As acknowledged in Appendix A, identity-disjoint splitting alone does not establish episode-level exchangeability, particularly when captions from the same identity may be correlated. Thus, the theoretical coverage guarantee is not fully justified by the evaluation protocol

  2. LAC and APS temperatures are optimized on the same calibration episodes used for conformal quantile estimation, potentially compromising their coverage validity

  3. The identity-level score uses the maximum image similarity, but its sensitivity to varying numbers of gallery images per identity is not investigated

  4. The evaluation assumes a fixed gallery and enrolled target identities; robustness under gallery updates, unseen identities, and distribution shifts remains unclea

Questions for authors

Please refer to the drawbacks

1 comment
Area_Chair_KhkiOpenReview ↗

The authors' rebuttal has been submitted

Dear reviewer, this is a kind reminder that the authors have posted their responses to your reviews. Please review their rebuttal, engage in any necessary follow-up discussion with the authors, and confirm whether your concerns have been addressed by updating your review/scores. Thank you for your colaboration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.