acceptodds
Withdrawn from ICLR 2027

Beyond Verbalized Feedback: Mining Intent-Aligned Visual Clues for Interactive Text-to-Image Person Re-Identification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image person re-identification (TIReID) aims to retrieve a target pedestrian from a large-scale image gallery using a natural-language description. Since a single description rarely captures the fine-grained details that distinguish visually similar pedestrians, interactive TIReID progressively completes the initial description through multi-round user feedback. However, existing methods rely on user feedback, and retrieved candidates are rarely used as a direct source of visual clues. As a result, subtle variations in color tone, texture, and clothing style, which are difficult to convey through textual feedback, can be easily overlooked. To address this issue, we propose Visual Clue-guided Interactive Retrieval (VCIR), a plug-and-play framework that mines intent-aligned visual clues from retrieved candidates. Specifically, we first design a Visual Clue Extraction (VCE) module that guides a Multi-modal Large Language Model (MLLM) to contrast these candidates against the user intent and extract discriminative clues. Then, we introduce a learnable retrieval token that distills these clues into a continuous representation, bypassing the information loss caused by verbalizing them into discrete text. We further embed VCE into a multi-round interactive retrieval framework, where the extracted clues directly serve as the retrieval query at each round and progressively steer retrieval toward the target pedestrian. VCIR introduces only 32.9M trainable parameters and easily integrates with existing interaction paradigms. Extensive experiments on Interactive-PEDES and MInterPEDES demonstrate that VCIR achieves state-of-the-art retrieval performance. The code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive Person Retrieval, Text-to-Image Person Re-identification, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cdo5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Visual Clue-guided Interactive Retrieval (VCIR) for interactive text-to-image person re-identification. VCIR uses a Visual Clue Extraction module to inspect the current Top-K candidate images together with the initial description and accumulated user feedback. A learned retrieval token collects intent-aligned visual clues, and a linear adapter maps its hidden state into the image-embedding space of a CLIP-based retriever. The resulting continuous query directly re-ranks the gallery at each round, avoiding the information loss of verbalizing all visual details. Training updates clue prompts, a retrieval token, an adapter, and LoRA parameters using per-round image-text contrastive and similarity-distribution-matching losses. Experiments cover Interactive-PEDES, MInterPEDES, and additional standard TIReID datasets.

Critical strengths
  1. The proposed continuous retrieval token effectively addresses the information loss caused by verbalizing fine-grained visual details into discrete text. By mapping directly into the CLIP image space, the method elegantly utilizes retrieved candidates as visual evidence in a plug-and-play manner without redundant text encoding.

  2. The framework features a well-designed, closed-loop system for multi-round retrieval. It seamlessly integrates continuous visual clues with textual feedback, applies supervision at every interaction step, and supports both chat-based and multimodal interaction formats robustly.

  3. The approach is highly parameter-efficient, training only 32.9M parameters while keeping the large vision and language backbones frozen. Furthermore, the paper provides rigorous and comprehensive evaluations across multiple datasets and interaction settings, utilizing trajectory-aware metrics (e.g., BRI) and thorough robustness analyses.

Critical weaknesses
  1. According to the introduction, the performance improvement is attributed to how the model "mines intent-aligned visual clues from retrieved candidates." However, Table 1 shows that the performance improvement mainly comes from MLLM query reconstruction and user feedback modeling rather than the proposed VCIR. It would be better to clarify the claim.

  2. The proposed method instructs the MLLM to "ignore appearance details in the candidate images that differ from the target person." But current experiments do not directly verify if the model actually filters wrong attributes effectively. Furthermore, regarding the statement that the method avoids "the information loss caused by verbalizing them into discrete text," there is no experimental comparison between the proposed retrieval token and discrete text representations. It would be better to provide a detailed analysis.

Questions for authors
  1. Appendix C shows that VCIR achieves good results on CUHK-PEDES, ICFG-PEDES, and RSTPReid. However, the compared baselines are limited. It is recommended to include more recent and representative TIReID methods for a fair comparison.
3 comments
AuthorsOpenReview ↗

Response to the Reviewer cdo5(1/2)

We thank the reviewer for recognizing our retrieval token, efficient design, and extensive evaluation. Your suggestions help clarify our contributions and strengthen the evidence. Below, we present additional experiments and corresponding revisions.

[W1] Candidate visual clues provide additional gains beyond MLLM query modeling.

Our claim concerns the additional benefit of extracting visual clues beyond this MLLM baseline, which we will state more clearly.

In Table 1, the baseline already uses the MLLM and accumulated user feedback but receives no candidate images. Compared with this baseline, VCIR improves R@1/mAP/mINP from 77.17%/60.06%/36.29% to 80.08%/70.00%/51.14%, yielding absolute gains of 2.91%/9.94%/14.85%. These gains are particularly pronounced in mAP and mINP, showing that candidate visual clues improve the ranking of relevant images beyond what textual feedback modeling achieves. Moreover, random candidate images yield only 59.91% mAP, compared with 70.00% using retrieved candidates, supporting the importance of relevant visual evidence.

We will revise the introduction to clarify that VCIR “mines intent-aligned visual clues from retrieved candidates to further improve MLLM-based retrieval.”

[W2] VCIR uses feedback effectively under conflicting visual evidence and outperforms explicit descriptions.

We thank the reviewer for identifying these two gaps in the experimental evidence. We address them with a controlled color-conflict experiment and an explicit description baseline.

Conflicting candidate details. We selected 1,318 samples from 1,203 queries in Interactive-PEDES. Each sample contains three non-target candidate images whose specified attribute colors conflict with the user feedback. We compare two settings: before interaction, the model receives the initial description and conflicting candidates; after interaction, it additionally receives a user feedback answer specifying a color inconsistent with all three candidates. Initial retrieval using the frozen pretrained CLIP text encoder provides an additional reference.

SettingR@1 (%)R@5 (%)R@10 (%)mAP (%)mINP (%)
Initial retrieval47.26970.41078.90737.99419.579
Conflicting candidates, before interaction53.64277.54283.00543.25922.646
Conflicting candidates, after interaction61.00282.09488.31648.47326.069

With the conflicting candidates unchanged, a single interaction improves R@1/mAP/mINP by 7.36%/5.21%/3.42%, providing behavioral evidence that VCIR uses feedback to improve retrieval despite misleading visual details.

Continuous embeddings versus explicit descriptions. We have added an explicit description baseline that shares VCIR’s model architecture and uses the same MLLM, candidate images, initial description, and user feedback. We modify the final part of the MLLM instruction: "Finally, describe the target person’s appearance in text for text-to-image retrieval." The resulting description is encoded by the pretrained CLIP text encoder. Results on Interactive-PEDES after 5 interaction rounds are reported below.

MethodR@1R@5R@10mAPmINPBRI ↓
Explicit description67.4283.3988.2851.7729.730.603
VCIR80.0888.4393.5470.0051.060.467

With the same visual and textual inputs, VCIR improves R@1, mAP, and mINP by 12.66%, 18.23%, and 21.33%, respectively. This comparison supports the advantage of continuous retrieval embeddings over generating explicit descriptions. We will add both experiments and clarify their purposes in Section 4.3.

AuthorsOpenReview ↗

Response to the Reviewer cdo5(2/2)

[Q1] VCIR retains the highest mAP across all three datasets with recent baselines included.

Following the reviewer’s suggestion, we expanded the comparison to include HAM [1] and ICL [2] from CVPR 2025, CECA [3] from CVPR 2026, and KPDM [4] from AAAI 2026.

MethodCUHK-PEDESICFG-PEDESRSTPReid
R@1R@5R@10mAPR@1R@5R@10mAPR@1R@5R@10mAP
IRRA73.3889.9393.7166.1063.4680.2585.8238.0660.2081.3088.2047.17
RDE75.9490.1494.1267.5667.6882.4787.3640.0665.3583.9589.9050.88
TBPS-CLIP73.5488.1992.3565.3865.0580.3485.4739.8361.9583.5588.7548.26
HAM77.7191.4294.5769.6868.2483.3088.1542.3071.6987.8593.3055.19
ICL79.0691.2694.7270.4470.0583.3587.9142.7072.5586.6091.3055.19
CECA79.6592.3595.2767.0774.3286.3390.0039.5673.4589.8093.4555.44
KPDM75.9790.7794.4668.1467.7882.4387.2640.3067.9584.6591.1051.88
VCIR85.5994.2296.3479.1469.7981.8386.3045.1976.1089.0592.4561.67

VCIR achieves the highest mAP on all three datasets and the highest R@1 on CUHK-PEDES and RSTPReid among the compared methods. CECA achieves higher R@1 on ICFG-PEDES, the Identity-Centric and Fine-Grained Person Description Dataset, which was specifically designed to provide detailed descriptions of identity-related attributes. These detailed initial descriptions leave less scope for additional information through interaction, which can help explain VCIR’s smaller advantage on this dataset. Nevertheless, VCIR achieves 45.19% mAP versus CECA’s 39.56%, demonstrating stronger overall ranking performance. We will update Appendix C with these comparisons and clarify the respective evaluation settings.

We thank the reviewer again for the constructive suggestions, which help us clarify VCIR’s contribution and strengthen the experimental evidence.

[1] Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification. CVPR 2025. [2] Human-Centered Interactive Learning via MLLMs for Text-to-Image Person Re-Identification. CVPR 2025. [3] Tackling Alignment Ambiguity in Person Retrieval through Conversational Attribute Mining. CVPR 2026. [4] KPDM: Key Phrase Dynamic Masking for Robust Text-to-Image Person Retrieval. AAAI 2026.

Reviewer_cdo5OpenReview ↗

Response to authors

Thank you for the additional clarifications and experiments. These have largely addressed my concerns. I'm willing to keep the score as of 3 (Weak Acceptance).

Reviewer_i3bGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Visual Clue-guided Interactive Retrieval (VCIR) framework for interactive text-to-image person re-identification. It uses a multimodal large language model to extract visual clues from top-ranked candidate images and incorporates them into iterative retrieval through learnable retrieval tokens. Experiments on interactive and standard benchmarks demonstrate competitive retrieval performance.

Critical strengths
  1. In VCIR framework, a intent-aware instruction is introduced, which cant force the MLLM to preserve confirmed attributes, emphasize new feedback, and ignore non-target details in candidates.
  2. The method is evaluated on multiple interactive and standard benchmarks, with comparisons to existing methods and experiments on different MLLMs.
Critical weaknesses
  1. The mechanism relies heavily on the instruction to ignore mismatched details. However, it lacks theoretical or qualitative proof on how effectively the model filters out noise without drifting toward false candidates.
  2. The contribution of intent-aligned visual clues is not convincingly demonstrated. In the candidate-image replacement ablation, R@1 changes only from 77.17% to 77.57%, while mAP slightly decreases from 60.06% to 59.91%. In addition, the simulated user feedback protocol and baseline comparisons require clearer explanations to establish experimental fairness.
Questions for authors
  1. Can you provide any qualitative or quantitative evidence that the token indeed encodes the fine-grained visual details that language misses?
  2. How sensitive is performance to the exact formulation of the three rules in the instruction? Have you tried ablations that remove or rephrase individual rules?
  3. In the robustness experiments, the feedback is generated from the clean description. Have you tried a more realistic setting in which the user only has the corrupted description and therefore may give inconsistent or incomplete feedback?
  4. What is the average end-to-end inference latency per interaction round compared to baseline methods (e.g., LLaVA-ReID)? Is the MLLM inference a bottleneck for real-time applications?
3 comments
AuthorsOpenReview ↗

Response to the Reviewer i3bG(1/3)

We thank the reviewer for recognizing VCIR’s instruction design and evaluation across datasets and backbones. Your suggestions help clarify the value of continuous visual representations, instruction sensitivity, feedback robustness, and inference cost. Below, we address each point with controlled experiments and additional measurements.

[Q1] Continuous visual clues representations outperform explicit textual descriptions.

The retrieval token ablation in Table 1 demonstrates its contribution to retrieval performance. Removing the retrieval token in Exp. (4) reduces R@1/mAP from 80.08%/70.00% to 78.71%/67.72%, corresponding to decreases of 1.37%/2.28%.

To directly examine the advantage over explicit language descriptions, we have added an explicit description baseline that shares VCIR’s model architecture and uses the same MLLM, candidate images, initial description, and user feedback. We modify the final part of the MLLM instruction: "Finally, describe the target person’s appearance in text for text-to-image retrieval." The resulting description is encoded by the pretrained CLIP text encoder, whereas VCIR directly produces a continuous retrieval embedding. Results on Interactive-PEDES after five interaction rounds are reported below.

MethodR@1 (%)R@5 (%)R@10 (%)mAP (%)mINP (%)BRI ↓
Explicit description67.4283.3988.2851.7729.730.603
VCIR80.0888.4393.5470.0051.060.467

With the same visual and textual inputs, VCIR improves R@1, mAP, and mINP by 12.66%, 18.23%, and 21.33%, respectively. Together with the token ablation, this comparison supports the retrieval token’s ability to capture useful visual information that is less effectively conveyed through explicit descriptions. We will add this comparison and clarify its connection to the token ablation in Section 4.3.

[Q2] VCIR maintains strong retrieval performance when any individual instruction rule is removed.

We thank the reviewer for suggesting an ablation of the individual instruction rules. We conducted three experiments, removing one rule at a time: (1) preserve confirmed target attributes; (2) emphasize user feedback; and (3) ignore candidate-image details inconsistent with the target.

MethodR@1 (%)mAP (%)mINP (%)
All three rules80.0870.0051.14
Without Rule 179.7969.7951.00
Without Rule 279.7069.8350.98
Without Rule 379.8169.6750.69
Without all three rules79.4868.8049.47

VCIR remains robust when any single rule is removed, with R@1/mAP decreasing by only 0.27%–0.38%/0.17%–0.33%. The preceding instruction already provides the overall task guidance, and these rules reinforce specific requirements within that guidance. Removing Rule 3 causes the largest individual decreases in mAP and mINP. Removing all three rules produces larger absolute decreases of 0.60%/1.20%/1.67% in R@1/mAP/mINP, demonstrating their combined benefit despite the modest effect of removing any single rule.

AuthorsOpenReview ↗

Response to the Reviewer i3bG(2/3)

[Q3] VCIR retains its advantage when feedback is inconsistent or incomplete from corrupted descriptions.

The reviewer’s suggestion prompted us to evaluate a more demanding setting in which the user feedback depends on the corrupted initial description. Specifically, we regenerate all 5 rounds of answers with only the corrupted initial description provided to the answerer. We use Qwen2.5-7B-Instruct, following the LLaVA-ReID answerer setup.

Results after 5 interaction rounds are reported as R@1 / mAP (%):

Initial descriptionHAMBaselineVCIR
Color words removed37.27 / 28.6756.34 / 43.1258.65 / 51.53
Second half truncated30.18 / 24.6247.24 / 38.0347.78 / 43.14
Fully uninformative21.63 / 18.0833.64 / 27.1232.82 / 29.85
All colors replaced18.31 / 15.5227.55 / 23.3332.84 / 30.14
Gender reversed48.42 / 38.4575.48 / 58.1577.72 / 68.44
Identity substituted19.71 / 19.5129.76 / 28.5433.17 / 34.31

Compared with the original robustness results in Table 4, inconsistent or incomplete feedback reduces VCIR’s average R@1/mAP across the six conditions from 47.53%/43.09% to 47.16%/42.90%. Nevertheless, VCIR retains the highest mAP across all six conditions and the highest R@1 in five. Its average R@1/mAP gains over HAM are 17.91%/18.76%, while candidate images contribute a further 6.52% average mAP gain over the baseline without images. These results support the value of extracting visual clues when the initial description is incomplete or misleading and the feedback is inconsistent or incomplete accordingly.

[Q4] VCIR achieves a favorable accuracy–efficiency trade-off.

We thank the reviewer for raising this practical question. We measure system inference latency under fixed feedback, including online processing and retrieval but excluding user/answer generation and offline preparation. All methods use the same QA pairs within each benchmark.

Interactive-PEDES. Both methods use 4× A800 GPUs (TP=1, batch size=1) . BF16 uses 16-bit floating-point model weights, while NF4 quantizes model weights to 4-bit NormalFloat to reduce GPU memory usage.

MethodWall time (s)Latency (ms/round)Peak memory (GiB)R@1 (%)mAP (%)mINP (%)
LLaVA-ReID (NF4 questioner)8,940.724,851.0010.9372.7453.3930.12
VCIR (BF16)931.32499.6418.6480.0870.0051.06
VCIR (NF4)1,244.02670.169.1478.1868.0449.09

Compared with LLaVA-ReID, VCIR in BF16 reduces mean measured latency by 4,351.36 ms and achieves absolute gains of 7.34%/16.61%/20.94% in R@1/mAP/mINP. NF4 offers an option for deployment with less GPU memory. Compared with VCIR in BF16, it reduces peak memory from 18.64 to 9.14 GiB, with latency increasing to 670.16 ms and R@1 decreasing by 1.90%. Compared with LLaVA-ReID, quantized VCIR requires 670.16 versus 4,851.00 ms and 9.14 versus 10.93 GiB, while maintaining absolute gains of 5.44%/14.65%/18.97% in R@1/mAP/mINP.

MInterPEDES. Both methods use InternVL2.5-1B on 1× A800 (TP=1, batch size=1). Latency includes initial retrieval, QA interaction and interaction with visual feedback.

MethodWall time (s)Initial retrieval (ms/round)QA (ms/round)Visual feedback (ms/round)R@1 (%)mAP (%)mINP (%)
MNEMO2,361.5433.9034.5358.3892.2380.3463.27
VCIR2,409.495.6152.7655.5493.4384.4870.75

With the same backbone, VCIR improves R@1/mAP/mINP by 1.20%/4.14%/7.48%, while complete-dialogue latency increases from 378.41 to 386.16 ms, adding only 7.75 ms. The QA interaction enables VCIR to examine candidate images alongside textual feedback, whereas MNEMO processes textual QA history alone. Visual feedback latency is lower at 55.54 versus 58.38 ms. Peak memory is 6.29 versus 3.24 GiB.

The measured results support VCIR for server-side interactive retrieval where accurate identification and ranking quality are priorities. We will add a subsection to the Appendix with the timing protocol, memory measurements, and stage-wise results to clarify the accuracy–efficiency trade-off.

AuthorsOpenReview ↗

Response to the Reviewer i3bG(3/3)

[W1] VCIR benefits from feedback despite conflicting visual clues.

We appreciate the reviewer’s concern about whether VCIR can resist misleading candidate details. To examine this directly, we conducted a controlled color-conflict experiment.

We selected 1,318 samples from 1,203 queries in Interactive-PEDES. Each sample contains three non-target candidate images whose specified attribute colors conflict with the user feedback. We compare two settings: before interaction, the model receives the initial description and conflicting candidates; after interaction, it additionally receives a user feedback answer specifying a color inconsistent with all three candidates. Initial retrieval using the frozen pretrained CLIP text encoder provides an additional reference.

SettingR@1 (%)R@5 (%)R@10 (%)mAP (%)mINP (%)
Initial retrieval47.26970.41078.90737.99419.579
Conflicting candidates, before interaction53.64277.54283.00543.25922.646
Conflicting candidates, after interaction61.00282.09488.31648.47326.069

With the conflicting candidates unchanged, a single interaction improves R@1/mAP/mINP by 7.36%/5.21%/3.42%, providing behavioral evidence that VCIR uses feedback to improve retrieval despite misleading visual details.

We hope this controlled experiment helps clarify how VCIR responds to conflicting candidate details. We thank the reviewer for motivating this analysis. We will add this experiment to the Appendix and discuss and analyze the results.

[W2] Intent-aligned visual clues improve retrieval beyond textual feedback.

We thank the reviewer for highlighting the need to clarify the candidate image ablation and experimental protocol. The improvement from intent-aligned visual clues is demonstrated by Exp. (6) versus Exp. (2) and Exp. (3), rather than by Exp. (3) versus Exp. (2). These experiments distinguish the effects of textual feedback modeling, random visual inputs, and relevant candidate images.

In Table 1, Exp. (2) shares VCIR’s architecture but receives no candidate images, with the image-related instructions removed. It measures the performance of modeling the initial description and user feedback through the MLLM alone. Exp. (3) supplies three randomly sampled gallery images, whose visual content may be unrelated to or inconsistent with the feedback. Its 0.40% R@1 increase and 0.15% mAP decrease relative to Exp. (2) show that random visual inputs provide little benefit while causing little overall degradation.

VCIR in Exp. (6) uses the three highest ranked candidates and achieves 80.08% R@1 and 70.00% mAP. It improves R@1/mAP by 2.91%/9.94% over the baseline without images and 2.51%/10.09% over the variant using random images. These gains show that the benefit comes from relevant visual clues extracted from candidate images, beyond textual feedback modeling or simply adding images.

Regarding the simulated user feedback protocol in Table 1, all compared methods are evaluated using the same QA pairs, obtained from LLaVA-ReID’s inference on Interactive-PEDES. This ensures that each method receives identical user feedback, allowing a controlled comparison of their retrieval performance. We will add a more detailed statement to Section 4.3: "All methods in Table 1 use the same QA pairs generated by LLaVA-ReID on Interactive-PEDES."

Reviewer_n8bfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VCIR for interactive text-to-image person re-identification. Starting from a text query, the method retrieves a small candidate set and uses it as visual input in the next interaction round. An MLLM receives these candidates, the initial description, accumulated user feedback, an intent-aware instruction, and learned clue prompts. A special retrieval token summarizes these inputs. Its final hidden state is projected into the embedding space of a frozen IRRA retriever to search the full gallery. The original MLLM weights remain frozen, while LoRA, the prompts, the retrieval token, and the projection adapter are trained. Contrastive and similarity distribution matching losses supervise each round. The framework supports both question-answering and feedback on reference images. Experiments on Interactive-PEDES and MInterPEDES report improved R@1 and mAP, with further ablations on candidate images, prompts, and the retrieval token.

Critical strengths
  1. The paper identifies a useful problem in interactive person retrieval. Retrieved candidates can provide visual information that the current description does not fully capture. Using this information to update the next query is a reasonable and practical direction.
  2. The retrieval results are promising. On Interactive-PEDES, VCIR reaches 75.06% R@1 and 66.13% mAP after three rounds, exceeding the corresponding five-round results of LLaVA-ReID. The improvements on MInterPEDES with InternVL-2.5 also provide useful evidence beyond changing the main MLLM backbone.
  3. The framework can reuse a frozen gallery encoder and its precomputed image features. Its query output is a single vector, and only a relatively small set of added parameters is trained. These are useful design choices for adapting an existing retrieval system.
Critical weaknesses
  1. The methodological contribution is not sufficiently distinguished from closely related work. CIR-LVLM [1] already combines task instructions, learned soft prompts, LoRA, and direct multimodal query embeddings. More directly, MNEMO [2] encodes reference images and feedback into continuous features and aggregates them with learned memory queries.
  2. The main claim about avoiding information loss from verbalization is not directly tested. The ablation without the retrieval token replaces it with the EOS hidden state. Both versions use continuous representations, so this comparison does not test the proposed benefit of avoiding textual descriptions. The model without candidate images also receives less information. A matched version that uses the same MLLM, candidates, and feedback to produce an explicit description is missing. Table 1 therefore supports the usefulness of the proposed representation, but does not isolate verbalization as the source of the performance gap.

[1] Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval⁠. AAAI 2025.

[2] Interactive Person Retrieval via Multi-Turn Multimodal Conversation⁠. ICML 2026.

Questions for authors
  1. What is the main technical difference from CIR-LVLM and MNEMO beyond the number and source of candidate images? Can VCE be compared with a standard joint image-text encoder using the same backbone, candidate inputs, feedback, gallery encoder, and adaptation budget?
  2. How are training candidates and trajectories constructed? At evaluation, do methods receive fixed shared feedback or feedback generated from their own candidates? Please specify the MInterPEDES round budgets and how early success is handled in later-round metrics and BRI. A short training algorithm and evaluation protocol table would help.
  3. A more detailed analysis on the efficiency should be discussed, such as training time, peak memory, per-round retrieval latency,
4 comments
AuthorsOpenReview ↗

Response to the Reviewer n8bf(1/3)

We thank the reviewer for recognizing VCIR’s use of visual clues, retrieval performance, and practical design. Your suggestions help strengthen our comparisons, experimental evidence, and evaluation reporting. Below, we address each concern with additional experiments and clarifications.

[W1, Q1] Technical differences from CIR-LVLM and MNEMO

VCIR and CIR-LVLM address different retrieval tasks and use different mechanisms to construct retrieval representations.

Differences from CIR-LVLM

Firstly, the tasks require different forms of reasoning. CIR-LVLM addresses composed image retrieval, where a relative caption specifies how a reference image should be modified to describe the desired target. VCIR identifies a specific person by progressively refining the query with user feedback. VCIR must preserve confirmed target attributes and extract additional clues consistent with the evolving description, while suppressing conflicting candidate details. Its objective is therefore to refine evidence about a target identity across interaction rounds.

Secondly, the methods aggregate multimodal information differently. CIR-LVLM uses position-weighted mean pooling over token features. VCIR employs a learnable retrieval token that gathers visual and textual information through attention, with its final hidden state forming the retrieval representation. This introduces a dedicated token trained to aggregate information.

To evaluate this representation design, we trained two joint image–text encoders using Qwen3-VL-8B. Both retain the same instructions, soft prompts, and LoRA, share an encoder between queries and gallery images, and are trained on 8× A800 GPUs under the same training budget. We compare position-weighted pooling with the retrieval token on Interactive-PEDES after 5 interaction rounds.

MethodTrainable parametersR@1 (%)R@5 (%)R@10 (%)mAP (%)mINP (%)BRI
Position-weighted pooling32,907,26476.2292.2395.7150.3023.670.4109
Retrieval token32,911,36078.1292.5596.0351.6324.620.3946

As shown in the results, the retrieval token increases R@1 from 76.22% to 78.12% and improves all other metrics, with only 4,096 additional parameters. These results support the effectiveness of learning a dedicated retrieval token for multimodal aggregation within this common encoder framework.

Differences from MNEMO

1. Candidate images play different roles. In MNEMO, the reference image supports user-provided corrective feedback and supplies the visual context needed to interpret it. The interaction centers on how the reference person differs from the target. VCIR treats retrieved candidates as a direct source of additional visual clues. It extracts appearance details consistent with the accumulated user intent, including details not explicitly expressed in the feedback, and incorporates them into the representation. As illustrated in Fig. 1, the distinction is between using images to contextualize corrective feedback and actively mining them for retrieval evidence.

2. VCIR can use multiple candidate images at every interaction round. MNEMO’s default protocol introduces one reference image in each of two visual feedback rounds. Its Appendix C reports that adding further visual feedback rounds increases inference time while providing limited accuracy gains. VCIR can inspect the current top-K candidates at every round, including the QA stage. This makes visual evidence available throughout the interaction. With the same InternVL2.5 backbone, VCIR improves R@1/mAP from 92.26%/80.31% to 93.64%/84.43% in Table 3, supporting the effectiveness of extracting intent-aligned clues from retrieved candidates.

We thank the reviewer again for this constructive suggestion. We will cite CIR-LVLM in the Related Work section and provide a detailed discussion of the technical distinctions between VCIR and both CIR-LVLM and MNEMO.

AuthorsOpenReview ↗

Response to the Reviewer n8bf(2/3)

[W2] Continuous visual clues representations outperform explicit textual descriptions.

To directly examine the advantage over explicit language descriptions, we have added an explicit description baseline that shares VCIR’s model architecture and uses the same MLLM, candidate images, initial description, and user feedback. We modify the final part of the MLLM instruction: "Finally, describe the target person’s appearance in text for text-to-image retrieval." The resulting description is encoded by the pretrained CLIP text encoder. Results on Interactive-PEDES after 5 interaction rounds are reported below.

MethodR@1R@5R@10mAPmINPBRI ↓
Explicit description67.4283.3988.2851.7729.730.603
VCIR80.0888.4393.5470.0051.060.467

With the same visual and textual inputs, VCIR improves R@1, mAP, and mINP by 12.66%, 18.23%, and 21.33%, respectively. This comparison supports the advantage of continuous retrieval embeddings over generating explicit descriptions. We will add this comparison and clarify the distinct purposes of these experiments in Section 4.3.

[Q2] Training trajectories and evaluation protocol.

At evaluation, all methods receive the same fixed feedback sequence. Answers are not regenerated from each method’s current candidates. We clarify the training procedure, interaction rounds, and metric computation below.

Training Algorithm.

Step 1. Encode the initial description with the frozen text encoder and retrieve the top-k candidate images.

Step 2. Add the next feedback in annotation order. Use the current candidates for QA feedback or the annotated reference image for visual feedback.

Step 3. Encode the initial description, accumulated feedback, and input images into a retrieval representation. Compute the ITC and SDM losses.

Step 4. Retrieve the next top-k candidates using the current query.

Step 5. Repeat Steps 2–4 until the available feedback is exhausted or the training round limit is reached. Accumulate gradients across rounds and update the trainable parameters.

At evaluation, all methods receive the same fixed feedback sequence. Our evaluation protocol is summarized in the table below.

ProtocolInteractive-PEDESMInterPEDES
Feedback sourceLLaVA-ReID QA trajectoriesDataset annotations
Feedback across methodsFixed and sharedFixed and shared
Interaction budget5 roundsUp to 17 rounds
QA rounds5Number of annotated QA pairs
Visual feedback rounds02

Early success and later rankings. Retrieving an image of the target identity at rank 1 does not terminate interaction or freeze subsequent results. Later R@K and mAP use the actual rankings, which may improve or decline. BRI uses the best target-identity rank observed so far, which remains 1 after the first success.

In summary, VCIR improves retrieval with both QA and visual feedback. All methods receive the same user feedback, ensuring a fair comparison. We will include the training algorithm and a more detailed evaluation protocol in the Appendix.

AuthorsOpenReview ↗

Response to the Reviewer n8bf(3/3)

[Q3] VCIR achieves a favorable accuracy–efficiency trade-off.

We thank the reviewer for raising this practical question. VCIR training takes 27 hours on 4× A800 GPUs. For inference, we measure system inference latency under fixed feedback, including online processing and retrieval but excluding user/answer generation and offline preparation. All methods use the same QA pairs within each benchmark.

Interactive-PEDES. Both methods use 4× A800 GPUs (TP=1, batch size=1) . BF16 uses 16-bit floating-point model weights, while NF4 quantizes model weights to 4-bit NormalFloat to reduce GPU memory usage.

MethodWall time (s)Latency (ms/round)Peak memory (GiB)R@1 (%)mAP (%)mINP (%)
LLaVA-ReID (NF4 questioner)8,940.724,851.0010.9372.7453.3930.12
VCIR (BF16)931.32499.6418.6480.0870.0051.06
VCIR (NF4)1,244.02670.169.1478.1868.0449.09

Compared with LLaVA-ReID, VCIR in BF16 reduces mean measured latency by 4,351.36 ms and achieves absolute gains of 7.34%/16.61%/20.94% in R@1/mAP/mINP. NF4 offers an option for deployment with less GPU memory. Compared with VCIR in BF16, it reduces peak memory from 18.64 to 9.14 GiB, with latency increasing to 670.16 ms and R@1 decreasing by 1.90%. Compared with LLaVA-ReID, quantized VCIR requires 670.16 versus 4,851.00 ms and 9.14 versus 10.93 GiB, while maintaining absolute gains of 5.44%/14.65%/18.97% in R@1/mAP/mINP.

MInterPEDES. Both methods use InternVL2.5-1B on 1× A800 (TP=1, batch size=1). Latency includes initial retrieval, QA interaction and interaction with visual feedback.

MethodWall time (s)Initial retrieval (ms/round)QA (ms/round)Visual feedback (ms/round)R@1 (%)mAP (%)mINP (%)
MNEMO2,361.5433.9034.5358.3892.2380.3463.27
VCIR2,409.495.6152.7655.5493.4384.4870.75

With the same backbone, VCIR improves R@1/mAP/mINP by 1.20%/4.14%/7.48%, while complete-dialogue latency increases from 378.41 to 386.16 ms, adding only 7.75 ms. The QA interaction enables VCIR to examine candidate images alongside textual feedback, whereas MNEMO processes textual QA history alone. Visual feedback latency is lower at 55.54 versus 58.38 ms. Peak memory is 6.29 versus 3.24 GiB.

The measured results support VCIR for server-side interactive retrieval where accurate identification and ranking quality are priorities. We will add a subsection to the Appendix with the timing protocol, memory measurements, and stage-wise results to clarify the accuracy–efficiency trade-off.

Thanks for the author's response which helps me to understand the paper. However, if we disregard this topic when evaluating this method, its contribution is somewhat limited, and this paper did not actually propose this topic. There have been similar papers before that performed interactive ReID, thus significantly decrease the paper's contribution. So, I will keep my score as 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.