acceptodds
Withdrawn from ICLR 2027

EAVI: Evidence-Aware Cross-Modal Alignment with Value-Guided Interaction for Chat-based Person Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chat-based person retrieval (ChatPR) identifies a target pedestrian in a gallery by progressively acquiring appearance evidence through multi-round question answering. This setting creates two requirements: a retriever must make effective use of evidence accumulated across turns, and a controller must acquire additional evidence selectively under a limited budget. We propose EAVI to address both. Its Dialogue Evidence Set Retriever (DESR) complements global matching with anonymous evidence sets aligned by entropy-regularized optimal transport and reliability-weighted aggregation, while keeping gallery representations cacheable. Its Value-Guided Interaction Agent (VGIA) selects from the recorded question pool using only the observable retrieval state, predicting each candidate's one-step retrieval value and deciding when to stop with an independent head. Because ranking questions within a dialogue and comparing values across dialogues rely on different properties of the value, VGIA treats question ordering, stopping, and shared-budget allocation as separate decisions. On ChatPedes, DESR improves R@1 by 1.7–2.4 points over the released DiaNA checkpoint under identical dialogue prefixes, with 3.0× fewer loaded parameters and about 6× faster full-test evaluation. A fitted question prior remains competitive for within-dialogue ordering, whereas retrieval-state feedback improves shared-budget allocation by 5.01 R@1 points on average over a question-and progress control across four budgets on a fixed acquisition path. A development-selected stopping head reduces additional QA acquisitions by 37.7% at a 0.67-point R@1 cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Chat-based person retrieval, interactive retrieval, cross-modal alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qSHxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies chat-based person retrieval, where a system progressively acquires appearance evidence through multi-turn question answering. The authors propose EAVI, which combines a Dialogue Evidence Set Retriever (DESR) with a Value-Guided Interaction Agent (VGIA). DESR augments global dialogue-image matching with slot-based evidence sets aligned via optimal transport, while VGIA selects questions from a recorded QA pool and decides when to stop based on the current retrieval state. Experiments on ChatPedes show modest retrieval gains over DiaNA, improved efficiency, and reduced interaction cost through adaptive stopping.

Critical strengths
  1. This paper focuses on an important variant of person retrieval, where the evidence acquisition is in a progressive chat-based setting.
  2. This paper provides a new perspective on disentangling the effects of evidence matching and evidence acquisition, and the proposed method follows the same paradigm.
  3. The proposed method is carefully designed, providing a reasonable fine-grained matching mechanism through anonymous evidence slots and optimal-transport-based alignment, while keeping gallery representations cacheable.
  4. Experiments show consistent improvements on the ChatPedes benchmark.
Critical weaknesses
  1. Citation and presentation issues. Some in-paragraph citations appear to use \cite rather than \citep, which leads to awkward formatting in the current draft.
  2. Concerns about the problem setup:
  • Limited realism of the pre-recorded QA protocol. The pre-recorded QA pool is useful for decoupling question selection from question generation and for analyzing different interaction-control abilities. However, it substantially simplifies the realistic ChatPR setting. In practical conversational retrieval, questions should be generated sequentially and conditioned on previously observed answers, rather than selected from a full pool of candidate questions available in advance.
  • Potential information leakage from future candidate questions. Even though the controller cannot observe unrevealed answers, seeing the full set of future candidate questions may leak useful information about the target. For example, if a candidate question is “Is the woman carrying a backpack?”, the question text itself already reveals gender-related information.
  • Limited evaluation scope. All main experiments are conducted on a single dataset, ChatPedes. This makes it difficult to assess the generality and soundness of the proposed method.
  1. Unclear contribution of some modules.
  • A fitted question prior performs as well as, or slightly better than, the full-state controller across all fixed ASK budgets. This weakens the claim that retrieval-state feedback is broadly useful for value-guided question selection.
  • The learned stopping head is not clearly stronger than a simple entropy-based rule, suggesting that the learned stopping module provides limited additional benefit in the current setting.
  • DESR is compared with DiaNA using substantially different dialogue encoder sizes. Therefore, the reported system-level efficiency gains may partly come from backbone/model-size differences rather than the proposed evidence-set alignment mechanism itself.
Questions for authors

Based on the claims made before, I think this paper is an analysis disentangling the effects of evidence matching and evidence acquisition, and is highly dedicated to a specific benchmark and setup, which limits the contribution of this paper (clearly below the acceptance threshold). The authors should answer the questions in the "weakness" section.

Reviewer_TTGPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EAVI, a chat based person retrieval framework that combines evidence aware cross modal alignment with value guided interaction control to improve retrieval accuracy and optimize question selection, stopping, and shared budget allocation.

Critical strengths

The paper addresses a meaningful interactive retrieval problem, presents a clear framework separating evidence matching from interaction control, and provides controlled experiments demonstrating improvements in retrieval accuracy, computational efficiency, and shared budget allocation.

Critical weaknesses

Thanks for submitting your paper, which is an insightful work. However, there are several concerns as follows:

  1. VGIA is motivated by using retrieval state information to guide evidence acquisition and interaction control. However, the Full State controller consistently underperforms the simpler Question Prior baseline in question ordering across all evaluated budgets. Similarly, the learned stopping head achieves nearly the same accuracy and interaction cost tradeoff as a simple entropy based rule. These results provide limited evidence that the proposed learned mechanisms offer meaningful benefits over simpler alternatives in these two tasks. The strongest gains from retrieval state information instead arise from shared budget allocation, whose practical relevance and representativeness for ChatPR require further justification.
  2. DESR shares several core design elements with existing fine grained cross modal retrieval methods, particularly PLOT (ECCV 2024), including slot based feature extraction, shared slot initialization, and weighted aggregation of local similarities. Its main distinctions lie in optimal transport based slot matching and reliability estimation. However, Table 2(a) shows only a marginal R@1 improvement from introducing slot extraction and OT without additional slot supervision. Furthermore, although Appendix A.6 reports a statistically supported improvement from the reliability gate, it does not establish that the gate mitigates evidence dilution. Thus, while DESR improves overall retrieval performance, the experiments provide limited evidence that its additional mechanisms specifically address the challenges motivating their design.
  3. VGIA relies on predicting the expected value of candidate questions, yet the evaluation primarily focuses on downstream retrieval performance without directly assessing the accuracy or calibration of these predictions. An analysis of value prediction accuracy, question ranking quality, and calibration across different retrieval states would help establish whether the proposed controller learns reliable question values and clarify the contribution of retrieval state information.
  4. All interaction experiments are conducted on ChatPedes using a fixed pool of recorded questions and answers. This setting may favor methods that exploit dataset specific question patterns and does not fully capture the variability of real user interactions. It therefore remains unclear whether the reported findings generalize to more diverse question distributions and open ended interaction settings.
Questions for authors
  1. Why does the Question Prior baseline consistently outperform the Full State controller in question ordering? Is this because question utility is largely determined by statistical regularities in ChatPedes, or because the proposed controller struggles to learn state conditioned question values? Would this finding persist under different question distributions or more diverse interactions?
  2. Why does retrieval state information substantially improve shared budget allocation but not question ordering within individual dialogues? Could the authors further explain why the same information has different effects on these two decisions?
  3. Could the authors further motivate the practical relevance of shared budget allocation in ChatPR? What realistic applications require multiple independent retrieval dialogues to share a global question budget, and how representative are the fixed acquisition paths used in the evaluation? This is particularly important since shared budget allocation is where retrieval state information demonstrates its most substantial gains.
  4. Could the authors provide a direct evaluation of the predicted question values beyond downstream retrieval performance? In particular, does the Full State controller improve prediction accuracy or calibration compared with Question Prior, even when it does not improve question ordering? Such an analysis would help clarify the contribution of retrieval state information. The questions above concern the main methodological contributions and their empirical justification. Clarifications or additional evidence addressing these concerns would help me reassess the paper during the rebuttal period.
  5. There are several minor LaTeX reference issues, such as "Equation equation 16" in Section 3.3 and "Eq. equation 14" in Section 4.2. These should be corrected.
Reviewer_kwjHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce EAVI, a chat-based person retrieval framework that pairs DESR, a dual-encoder retriever with optimal-transport alignment of cached evidence slots, with VGIA, a controller that picks questions from a recorded QA pool by predicted one-step value and uses a separate head to decide when to stop. They evaluate on ChatPedes.

Critical strengths
  1. DESR is a better and much cheaper ChatPR retriever than the prior state of the art. Even if the gain is mostly due to the backbone rather than CAET (W1), a smaller, faster, more accurate retriever is a useful result.
Critical weaknesses

W1. Issues with the results

  1. Looking at the description of recorded order and fixed prefix, it seems that the following results should be identical, but are not: K=3 and r04, K=5 and r06, and K=7 and r08. Table 5 matches the fixed-prefix numbers, but not the recorded order. If the fixed-prefix numbers are right, the full EAVI system at K=7 does worse than just reading the first eight recorded turns.
  2. VGIA doesn’t beat a simple question prior for ordering. In Table 1(b), the full EAVI system is below the question prior at every K, and in Table 5 the full-state controller is below even the question-only arm. The ~10-point gain over recorded order comes from a dataset-level count model, not from value-guided interaction.
  3. The learned stopping head is essentially tied with a single entropy threshold. The preregistered stopping rule, which used VGIA’s value head, missed its tolerance on test, and the reported configuration was chosen afterward.
  4. Shared-budget allocation is the only mode where retrieval state helps, but it’s also the least motivated setting. It’s unclear why dialogues from different users would share a single question budget in deployment. The headline +5.01 is also measured against question+progress, which is weaker than question-only; against question-only, the gain is 4.45.
  5. DESR’s novel components contribute little. In Table 2(a), slot extraction with OT adds +0.18 R@1, within noise, and the two forms of slot supervision give identical results, so the transport formulation itself doesn’t seem to matter. The gain over DiaNA also comes with a different, smaller backbone and DiaNA’s own augmentation strategy, so without a same-backbone comparison it can’t be attributed to CAET.

W2. The evaluation doesn’t test the setting the paper claims to address.

  1. ChatPedes questions were generated by an LLM talking to a simulated user who sees the target image, so later questions are follow-ups on earlier answers (e.g., “Can you describe more detail about the shorts?”). VGIA hashes the question text into its descriptor, so the controller is target-blind in its inputs but not in its action space. No deployed system would have candidate questions written with knowledge of the target.
  2. Reordering a recorded dialogue produces dialogues that could never happen. A follow-up can be revealed before the answer that prompted it. Since acquired pairs are re-serialized in original order anyway, “question ordering” is really choosing a subset of a target-specific pool.
  3. The controller’s state relies on ground-truth identity labels at inference to pool image scores into identity scores. The paper argues this would correspond to track-level association in deployment, but it’s still oracle grouping that the experiments don’t replace with anything realistic.
Questions for authors
  1. Can you explain your results pointed at in W1.1? I'm not sure if the paper is not clear or it is an actual issue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.