acceptodds
Under review as a conference paper at ICLR 2027

ARK: A Benchmark for Reasoning Re-Identification in Multimodal Large Language Models

Abstract

Re-identification (ReID) is commonly formulated as a retrieval problem, yet ranking alone does not always resolve identity: an observation of the correct identity may appear among the top-k candidates without being ranked first, while open-set queries require recognizing when no valid match exists. We formulate this unresolved post-retrieval stage as Reasoning ReID, where a multimodal large language model (MLLM) verifies retrieved candidates before producing the final identity decision. To systematically study the capabilities required for such verification, we introduce ARK, a benchmark for Reasoning ReID using Animal ReID as a challenging testbed. ARK spans 17 species and seven controlled protocols organized into three capability families: Visual Identity Perception, Evidence Integration & Association, and Decision Reliability. An evaluation of 21 proprietary and open-weight MLLMs alongside a human reference shows that current models can benefit from complementary evidence, while their reliability remains uneven under demanding verification conditions. We further instantiate Reasoning ReID in retrieval pipelines built on CLIP and SigLIP, showing that selectively invoking MLLM verification for uncertain retrievals can improve end-to-end identification in both closed- and open-set settings.

Primary area:
datasets and benchmarks
Keywords:
Animal ReID, Multimodal Large Language Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jb73OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper formulates Reasoning ReID, a post-retrieval MLLM verification stage, and introduces ARK, an Animal ReID benchmark of 187,865 evaluation instances across 17 species and seven diagnostic protocols organized into three capability families. An evaluation of 21 MLLMs alongside a human reference shows that models can benefit from complementary evidence, while their reliability remains uneven under demanding verification conditions.

Critical strengths
  1. Beyond single-match accuracy, ARK evaluates multi-match identity association, open-set rejection, and resistance to misleading suggestions, which retrieval metrics alone do not capture. For example, P5 shows distinct precision–recall profiles: Qwen3.5-122B is balanced (71.38% precision, 67.71% recall), whereas Gemma3-27B over-selects (73.70% recall, 49.77% precision).
  2. Controlled diagnostic design. ARK holds candidate sets fixed across corruption, multi-view, and context conditions, and image pairs fixed across prompt conditions. These controls help distinguish the effects of the evaluated conditions from changes in candidate composition.
Critical weaknesses
  1. Decision Reliability needs further diagnosis. Proprietary models achieve 43.64–53.49% accuracy on P1 but only 4.21–5.54% AccMP on P6, and 47.93–48.14% AccMA. Because the parsing audit covers only P1, the source of this drop remains unresolved. In P7, LLaVA-13B’s leading AccM among the models shown accompanies chance-level AccN, so paired Neutral/Misleading predictions are needed to measure how often correct Neutral judgments are preserved under misleading prompts.
  2. Limited validation of pipeline gains. Routed-MLLM improves closed-set accuracy by around one percentage point, while Always-MLLM underperforms Retrieval-only. Only one verifier is evaluated, requiring approximately 40 seconds per verification call, and the routing frequency and average Routed-MLLM latency are unreported.
Questions for authors
  1. In the P3 comparison with N = 3 (Fig. 3b), what fraction of instances lack a majority under Vote for Qwen3-VL-8B, Qwen3.5-35B, and Qwen3.5-122B? Does Joint still outperform Vote when ties are resolved uniformly at random among the tied predictions, measured as expected accuracy?
  2. Key information needed to interpret the main results appears only in the appendix, such as the Routed-MLLM routing rule.
  3. The presentation is overextended and could be more focused. Extensive procedural descriptions occupy substantial space, while some results remain insufficiently explained, such as unexpected Decision Reliability results.
4 comments
AuthorsOpenReview ↗

Responses to Reviewer Jb73

W1. Decision Reliability: Open-Set Rejection and Resistance to Misleading Suggestions

To investigate the unusually low P6 match-present accuracy, we add a rejection-behavior analysis of three proprietary MLLMs and Qwen3.5-122B across 24,314 instances (Appendix G.2). The proprietary models incorrectly select "None of the above" on 87.52–88.83% of match-present instances, despite a valid match being available. These are parseable but incorrect decisions, demonstrating that excessive rejection, rather than output-format failures, is the primary explanation for their low AccMP (4.21–5.54%). The three proprietary models also achieve 99.96–99.99% first-response parsing validity in our P1 audit (Appendix G.1).

We have added a paired analysis of LLaVA-13B, Qwen3.5-122B, and Claude-Opus-4.6 using identical image pairs under Neutral and Misleading prompts (Appendix G.4). We report four correctness transitions and Correctness Preservation, defined as the proportion of neutral-correct judgments that remain correct under misleading prompts, macro-averaged across 11 species. The results show that strong performance under Neutral prompts does not guarantee resistance to misleading suggestions. Qwen3.5-122B and Claude-Opus-4.6 achieve 72.66% and 73.52% AccN, but 67.51% and 69.83% of all pairs transition from correct to incorrect, with correctness preservation rates of only 6.85% and 4.85%, respectively. In contrast, LLaVA-13B achieves relatively high AccM (44.84%) and preservation (74.55%). However, its AccN is already near chance (49.49%), and 42.57% of pairs remain incorrect under both prompts. Its relatively high AccM and preservation therefore do not establish reliable identity verification.

W2. Pipeline Gains, Verifier Generality, and Inference Efficiency

Our retrieval-derived evaluation assesses the value of MLLM verification under both closed-set identification and open-set shortlist rejection. With Qwen3.5-122B, Routed-MLLM improves closed-set Rank-1 accuracy over Retrieval-only by 1.02 and 1.12 percentage points for CLIP and SigLIP, respectively, while increasing open-set BAcc over Retrieval+Threshold by 4.36 and 4.37 points (Section 4.4; Appendix D.4). These larger open-set gains highlight the value of MLLM verification for shortlist rejection beyond conventional closed-set ranking.

To further assess verifier generality, we extend the evaluation to four verifiers: Qwen3.5-122B, Qwen3.5-35B, Qwen3-VL-235B, and Claude-Opus-4.6 (Appendix D.8). Routed-MLLM improves closed-set Rank-1 accuracy over Retrieval-only by 0.09–1.23 percentage points and open-set BAcc over Retrieval+Threshold by 2.87–4.37 points. Moreover, Routed-MLLM consistently outperforms Always-MLLM across all four verifiers and both retrievers, supporting selective rather than uniform MLLM verification.

Regarding inference efficiency, an individual verification call with Qwen3.5-122B takes approximately 40 seconds. At the selected threshold, only 21.19% and 21.16% of queries invoke the verifier for CLIP and SigLIP, respectively. This yields average latencies of 8.26 s and 9.00 s under closed-set identification, and 9.58 s and 9.51 s under open-set evaluation (Appendix D.5–D.6). Our routing-threshold analysis further shows that increasing verification frequency generally raises latency without improving closed-set accuracy, supporting selective routing as a more favorable accuracy–latency trade-off than full verification.

Q1. Majority-Vote Tie-Breaking Sensitivity

Thank you for this suggestion. We examine no-majority cases in P3 (N=3N=3) and evaluate uniform random tie-breaking, with expected accuracy computed analytically. The results are summarized below.

ModelNo Majority (%)Strict Vote (%)Random-Tie Vote (%)Joint (%)Δ (pp)
Qwen3-VL-8B18.635.0836.6337.20+0.57
Qwen3.5-35B14.948.5650.6551.96+1.31
Qwen3.5-122B12.455.4757.8359.44+1.61

No Majority is pooled; accuracies are species-macro averaged. Δ = Joint − Random-Tie Vote.

Joint continues to outperform Random-Tie Vote across all three models, confirming that its advantage is not solely attributable to the treatment of no-majority cases. We have added the detailed results and analysis to Appendix G.7.

Q2. Routing Clarity in the Main Paper

We have revised Section 4.4 to clarify how Routed-MLLM identifies uncertain retrievals. Retrieval uncertainty is measured using the margin between the top-1 and top-2 retrieval scores, normalized to a species-specific percentile based on validation data. Lower percentiles indicate greater retrieval uncertainty, and queries falling below a validation-selected threshold are routed to the MLLM verifier.

Q3. Presentation Focus and Result Interpretation

We have strengthened the main-text discussion by providing clearer explanations of the Decision Reliability findings. For P6, our new rejection-behavior analysis identifies excessive selection of "None of the above" as the primary explanation for the proprietary models' low match-present accuracy (Appendix G.2). For P7, the paired Neutral–Misleading analysis provides a more detailed understanding of model robustness by distinguishing the preservation of correct judgments from accuracy under individual prompting conditions (Appendix G.4). These findings are now incorporated into the main Results discussion, providing a clearer interpretation of the observed performance patterns.

Thank you for the detailed rebuttal.

W2. While the authors extend the evaluation to four verifiers, the closed-set gains are small, and most gains being marginal. Moreover, the gains show no consistent relationship with model size and appear somewhat arbitrary, making it unclear when or why MLLM verification helps. Considering the substantial inference latency introduced by the verifiers, the practical significance of these improvements are limited. Although the open-set results are much better, my concern about the practical value of MLLM verification and its connection to the benchmark's evaluated capabilities remains.

Therefore, I will maintain my original score.

Thank you for the follow-up. We would like to further clarify the patterns in the cross-verifier results and the contribution of ARK.

Verifier Performance. Appendix D.8 shows consistent patterns across both retrievers. All four verifiers achieve positive gains with selective routing, and their performance rankings are identical between CLIP and SigLIP within each evaluation setting. Within the Qwen3.5 family, scaling from 35B to 122B improves closed-set gains from 0.49–0.55 to 1.02–1.12 percentage points, and open-set gains from 2.87 to 4.36–4.37 points. Across different model families, parameter count alone is an insufficient predictor of performance. For example, despite similar nominal parameter counts, Qwen3-VL-30B substantially outperforms Gemma3-27B on P1 (36.96% vs. 24.83%) and P3 (39.79% vs. 24.25%), as shown in Table 1. Furthermore, Routed-MLLM consistently outperforms Always-MLLM across all four verifiers and both retrievers, highlighting the importance of selective verification.

Benchmark Contribution and Practical Value. ARK aims to systematically characterize the capabilities and limitations of general-purpose MLLMs as identity verifiers. Through seven controlled protocols and an evaluation of 21 MLLMs, ARK investigates visual identity perception, evidence integration and association, and decision reliability. Section 4.4 and Appendix D.8 further examine these verifiers in retrieval-based pipelines. The modest closed-set gains, stronger open-set improvements, and substantial inference costs reveal both the potential and current limitations of MLLM verification. These findings provide insights into verifier behavior and highlight challenges for future MLLM-based ReID research.

Reviewer_vn6NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ARK, a benchmark for Reasoning Re-Identification (ReID), which evaluates the ability of multimodal large language models (MLLMs) to verify identities beyond conventional retrieval-based ranking. ARK covers 17 animal species and seven protocols spanning visual identity perception, evidence integration, and decision reliability. The authors evaluate 21 MLLMs and further demonstrate the practical utility of selective MLLM verification in retrieval-based ReID pipelines.

Critical strengths
  1. The paper is well-motivated and addresses an important limitation of conventional retrieval-based ReID by focusing on identity verification beyond similarity ranking.
  2. The benchmark is comprehensive and carefully constructed, covering diverse capabilities across 17 species and seven evaluation protocols.
  3. The experiments are extensive and reveal interesting findings, particularly the benefits of multi-view evidence and the weaknesses of MLLMs in open-set rejection and misleading-prompt settings.
Critical weaknesses
  1. Most benchmark protocols use randomly sampled same-species distractors rather than retrieval-generated hard negatives, which may limit their relevance to practical ReID scenarios.
  2. The end-to-end closed-set improvements are relatively modest (around 1 percentage point), despite the substantial inference cost. Comparisons with conventional ReID re-ranking methods are missing.
  3. Potential pretraining contamination in P4 and the substantial impact of output parsing retries on some models raise concerns about the reliability of cross-model comparisons.
Questions for authors
  1. How well do the benchmark findings generalize to retrieval-generated candidate sets with harder negatives?
  2. How does the proposed MLLM verification compare with conventional visual re-ranking methods in terms of accuracy and efficiency?
  3. What proportion of queries is routed to the MLLM, and how does the accuracy–latency trade-off vary with the routing threshold?
  4. Could the authors further investigate potential pretraining contamination in P4 and the effects of output parsing retries on model comparisons?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer vn6N

W1. Controlled Distractors vs. Retrieval-generated Hard Negatives

ARK deliberately uses controlled same-species sampling to evaluate MLLM identity verification independently of any particular retriever. Retrieval-generated hard negatives depend on the representation and ranking behavior of the chosen retriever, making the evaluation results partly dependent on that retriever. Our controlled construction instead provides consistent candidate sampling across MLLMs, allowing us to isolate and compare their identity-verification capabilities.

To complement this diagnostic evaluation, we also evaluate MLLM verification on actual top-5 candidate shortlists generated by CLIP and SigLIP, covering both closed-set identification and open-set rejection (Section 4.4). This provides practical validation under retrieval-generated candidate distributions while preserving the retriever-independent design of ARK's diagnostic protocols.

W2. Limited End-to-End Gains and Comparison with Conventional Re-ranking

To address this concern, we have added a new comparison with four visual post-processing baselines: k-Reciprocal Re-ranking (KR) [1], Expanded Cross Neighborhood (ECN) [2], Graph Convolution Re-ranking (GCR) [3], and Pose2ID-NFC [4] (Appendix D.7). We adapt their official implementations to a candidate-constrained setting, where all methods use the same CLIP- or SigLIP-generated top-5 shortlists. The visual baselines operate only on the corresponding query and candidate embeddings, ensuring consistent candidate access across methods.

MethodCLIP (%)SigLIP (%)
Retrieval-only23.1324.38
KR22.6525.12
ECN21.9224.76
GCR21.9524.26
Pose2ID-NFC22.8524.72
Routed-MLLM24.1525.50

Routed-MLLM achieves the highest species-macro Rank-1 accuracy with both retrievers, demonstrating competitive performance against conventional visual re-ranking methods. More importantly, closed-set ranking captures only one aspect of Reasoning ReID. Our retrieval-derived validation also demonstrates stronger open-set rejection than calibrated retrieval thresholding, improving balanced accuracy from 43.96% to 48.32% with CLIP and from 44.99% to 49.36% with SigLIP (Section 4.4; Appendix D.4). ARK's diagnostic protocols further examine multi-view evidence integration and context-assisted verification.

Regarding inference cost, selective routing invokes the MLLM for approximately 21% of queries, with average closed-set latency of 8.26 s for CLIP and 9.00 s for SigLIP (Appendix D.5–D.6). Routing therefore concentrates MLLM computation on uncertain retrievals, avoiding verification for approximately 79% of queries.

References

[1] - 2017 - CVPR - Re-Ranking Person Re-Identification with k-Reciprocal Encoding.

[2] - 2018 - CVPR - A Pose-Sensitive Embedding for Person Re-Identification with Expanded Cross Neighborhood Re-Ranking.

[3] - 2024 - IEEE TMM - Graph Convolution Based Efficient Re-Ranking for Visual Retrieval.

[4] - 2025 - CVPR - From Poses to Identity: Training-Free Person Re-Identification via Feature Centralization.

W3. Pretraining Contamination and Output Parsing Reliability

To assess the contribution of contextual evidence in P4 beyond the unusually high image-only scores of proprietary models, we examine three open-weight Qwen3.5 models under controlled input conditions. Adding full context improves accuracy over image-only input by 5.46, 7.45, and 9.10 percentage points for Qwen3.5-9B, 35B, and 122B, respectively, while metadata-only performance remains near chance. These consistent gains support the contribution of contextual evidence to identity verification across different model scales (Section 4.3, Figure 3(c)). We also discuss potential pretraining exposure associated with publicly available MetaWild images in Appendix A.4.

As a diagnostic benchmark, ARK aims to assess MLLM identity-verification capabilities while minimizing the interference of output-format failures. Related MLLM benchmarks also employ answer extraction, fallback procedures, or retry settings to handle model outputs [1][2][3]. ARK retries only when an output is unparsable, using the same prompt and decoding configuration, and scores the first parseable response regardless of correctness. This allows models to recover from formatting failures without introducing correctness-based answer selection or ground-truth feedback.

We further report strict first-response accuracy, parsing validity, and recovery statistics for 11 MLLMs on P1 (Appendix G.1, Table 15). The within-family scaling trends remain consistent under strict-first evaluation, and the additional analysis makes the effects of recovery explicit, allowing readers to distinguish identity-verification performance from output-format reliability.

References

[1] - 2024 - ECCV - BLINK: Multimodal Large Language Models Can See but Not Perceive.

[2] - 2025 - ICLR - MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding.

[3] - 2026 - ICLR - MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence.

Q1. Generalization to Retrieval-generated Hard Negatives

We evaluate generalization using actual top-5 candidate shortlists retrieved by CLIP and SigLIP across 11 species (Section 4.4; Appendix D). Unlike random distractors, these shortlists contain high-similarity negatives selected by the retrievers, providing a direct evaluation under retriever-specific hard-negative conditions. Under these conditions, Routed-MLLM improves closed-set Rank-1 accuracy over Retrieval-only and open-set balanced accuracy over calibrated thresholding for both retrievers. These results demonstrate the practical utility of MLLM verification on retrieval-generated candidate sets.

Q2. Accuracy and Efficiency Compared with Conventional Re-ranking

Our additional experiments (Appendix D.7) show that Routed-MLLM achieves the highest closed-set Rank-1 accuracy with CLIP (24.15%) and SigLIP (25.50%), compared with 22.85% and 25.12% for the strongest conventional re-ranking baselines, respectively.

Regarding efficiency, conventional re-ranking operates on precomputed visual embeddings, whereas MLLM verification incurs additional inference cost. To manage this cost, Routed-MLLM invokes the verifier for approximately 21% of queries, achieving average end-to-end latencies of 8.26 s and 9.00 s for CLIP and SigLIP, respectively (Appendix D.5–D.6). This avoids MLLM verification for approximately 79% of queries.

Q3. Routing Frequency and Accuracy–Latency Trade-off

We thank the reviewer for this suggestion. We have added a routing-frequency and accuracy–latency analysis using CLIP and SigLIP with Qwen3.5-122B (Appendix D.6). At the validation-selected threshold ρ=0.2\rho = 0.2, 21.19% and 21.16% of queries are routed to the MLLM, yielding average latencies of 8.26 s and 9.00 s under closed-set identification, and 9.58 s and 9.51 s under open-set evaluation, respectively. We further sweep the routing threshold from 0.0 to 0.9 in increments of 0.1 under closed-set identification, with Always-MLLM as the full-routing endpoint. Both retrievers achieve their highest Rank-1 accuracy at ρ=0.2\rho = 0.2, reaching 24.15% and 25.50%, respectively. Increasing the threshold further generally raises latency without improving accuracy. At full routing, accuracy decreases to 20.60% and 22.47%, respectively. These results show that more frequent MLLM verification does not necessarily improve identification performance, supporting selective routing as a more favorable accuracy–latency trade-off in this setting.

Q4. Pretraining Contamination and Parsing Retry Effects

For P4, we further evaluate three open-weight Qwen3.5 models to assess contextual gains independently of the proprietary-model results. Full context improves accuracy over image-only input by 5.46–9.10 percentage points, while metadata-only performance remains near chance (Section 4.3, Figure 3(c)). We also discuss potential pretraining exposure in Appendix A.4.

For output parsing, we add a strict first-response analysis of 11 MLLMs (Appendix G.1, Table 15). The analysis quantifies the impact of parsing recovery and confirms that within-family scaling trends remain consistent under strict-first evaluation. By reporting both strict-first and recovered accuracy, we provide a more transparent and comprehensive basis for model comparisons.

Reviewer_u4puOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors introduce ARK, a benchmark for evaluating whether multimodal large language models can identify the same individual animal across different images. It covers 17 species and seven protocols which include identity matching, image corruption, multiple query views, contextual information, multiple valid matches, rejection when no match exists, and resistance to misleading prompts. The authors also evaluate 21 models and alongside a human reference. They also investigate a retrieval pipeline that selectively uses an MLLM to verify uncertain matches.

Critical strengths
  • Individual animal identification requires distinguishing subtle visual features despite changes in pose, lighting, and background. Evaluating candidate verification, multiple matches, and rejection is useful for wildlife monitoring and complements conventional retrieval benchmarks.
  • The paper tests corruption, candidate-set size, conflicting views, precision–recall behavior, output recovery, and identity weighting. Separating match-present accuracy from match-absent rejection reveals failures that an overall score can often hide.
  • Effort has gone into the benchmark construction. Identity sampling caps, preference for previously unused queries, and distractor-overlap controls help reduce repetition. The source mappings, preprocessing specifications, and fingerprint checks provide a useful details for reconstruction.
Critical weaknesses
  • My main concern is that ARK combines existing animal ReID datasets with tasks whose underlying settings have substantial prior work. Market-1501 studies multiple-query identification and multiple correct gallery matches. MetaWild already evaluates metadata-assisted animal identification. WildlifeReID-10k includes closed-set and open-set animal identification. Combining these settings into an MLLM evaluation suite can be useful, but the paper does not clearly establish what new scientific understanding this combination provides.
  • The abstract states that ARK enables systematic study of the capabilities required for reasoning-based verification. But I think that the connection between this claim and the benchmark design is not established sufficiently. The authors reasonably motivate animal identification through subtle individual differences and the absence of clothing cues, but what specific properties of ARK allow it to diagnose reasoning beyond visual identity matching? I could use existing ReID datasets which contain different observations of the same individual and visually similar alternatives. Many of ARK’s protocols could therefore be constructed by changing the questions, candidate sets, or input conditions on these datasets. The authors should discuss what additional insight Ark provides beyond applying similar MLLM evaluation protocols to existing ReID benchmarks.
  • The claim “We formulate this unresolved post-retrieval stage as Reasoning ReID” also requires clearer positioning against prior work. Works like LamRA already retrieves candidates using embedding similarity and then applies an MLLM to rerank or select among them. NeuralCatalog also combines encoder-based retrieval with a final MLLM decision for animal recognition.

Liu et al., LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Khan et al., Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generation

  • Always invoking the verifier reduces closed-set accuracy relative to retrieval alone. Selective routing improves accuracy by very little margins.
Questions for authors
  • Can the authors comment more on the huge gap in closed models for the AccMP and AccMA in Table 1. It is surprising that closed sourced models are that bad on AccMP. WHy do they this happens? The results do not look intuitive to me.
  • The verification latency of approximately 40 seconds per query also makes the routed fraction and overall system latency important, What percentage of queries need this slow MLLM check?
  • A table comparing existing benchmark in the ReID field with ARK should be present in the paper. Makes it much easier to compare the contribution of ark.
5 comments
AuthorsOpenReview ↗

Responses to Reviewer u4pu

W1. Evaluation Target and Scientific Novelty

The cited ReID benchmarks primarily evaluate retrieval and ranking performance, whereas ARK evaluates the ability of general-purpose MLLMs to verify individual identities among candidate observations. Although some evaluation settings appear similar, they address different questions. For example, multi-query retrieval evaluates how multiple observations improve identity ranking, while ARK-P3 examines whether an MLLM can integrate evidence across query views to identify a matching candidate. Likewise, ARK-P6 evaluates the model's explicit decision to select a matching candidate or reject the entire candidate set.

ARK also reveals limitations of MLLM identity verification that conventional retrieval metrics do not capture. For example, Claude-Opus-4.6 achieves only 5.54% match-present accuracy versus 48.14% match-absent accuracy on P6. On P7, its accuracy drops from 73.52% under neutral prompts to 4.61% under misleading prompts, despite evaluating the same image pairs. More broadly, our cross-protocol analysis of 21 MLLMs (Appendix G.6) shows that P6 match-present accuracy correlates only weakly with P1–P3 accuracy (r = 0.18–0.20), while P7 accuracy under misleading prompts is negatively correlated with P1–P3 (r = −0.86 to −0.89). These findings indicate that strong performance in standard identity matching does not necessarily translate into reliable open-set decisions or robustness to misleading suggestions.

W2. Motivation and Validity of Reasoning ReID

ARK evaluates MLLM identity verification through seven protocols covering three capability families: Visual Identity Perception (P1–P2), Evidence Integration & Association (P3–P5), and Decision Reliability (P6–P7). Specifically, P1–P2 examine fine-grained identity matching under clean and corrupted visual evidence; P3–P4 assess the integration of multiple views and auxiliary context; P5 requires identifying all valid matches; and P6–P7 evaluate rejection decisions and sensitivity to misleading suggestions. These protocols use controlled comparisons to examine different aspects of identity verification. For example, P7 keeps the image pair fixed while varying the textual suggestion, allowing its effect on the identity decision to be measured.

ARK is not limited to the current dataset collection. We provide a unified task formulation, standardized diagnostic protocols, and reusable construction and evaluation scripts that can be adapted to additional ReID datasets. Researchers can apply the same protocols and metrics to evaluate MLLM identity verification across different species and datasets.

W3. Positioning Against LamRA and Neural Catalog

LamRA and Neural Catalog both incorporate LMMs/MLLMs into retrieval pipelines. LamRA trains LMMs for general multimodal retrieval and pointwise/listwise relevance reranking, while Neural Catalog combines retrieval and visual reranking with MLLM-based species recognition.

ARK focuses on a different aspect of this general approach: evaluating the ability of general-purpose MLLMs to verify individual identities among candidate observations. Its seven controlled protocols examine fine-grained identity perception, evidence integration, identity association, and decision reliability. These capabilities are evaluated independently of any particular retriever, with additional experiments using CLIP- and SigLIP-generated shortlists to assess their relevance to practical ReID pipelines.

We have added both studies to the Related Work section to clarify their relationship to ARK.

W4. Retrieval-Derived Validation and Performance Gains

The retrieval-derived experiments evaluate MLLM verification on candidate shortlists produced by actual retrievers, rather than proposing a new ReID method. Their purpose is to examine whether MLLMs can improve identity decisions beyond retrieval-based ranking.

The results suggest that MLLM verification can complement retrieval when applied selectively. CLIP and SigLIP achieve Rank-5 recall of 42.44% and 44.27%, respectively, leaving considerable room for improving candidate selection beyond Rank-1 retrieval. While Always-MLLM reduces closed-set accuracy, selective routing provides improvements of 1.02 and 1.12 percentage points. The benefits are more pronounced in the open-set setting, where routing improves balanced accuracy by up to 4.37 percentage points over calibrated retrieval thresholding, with gains of up to 5.73 percentage points in match-present accuracy and 3.60 percentage points in match-absent rejection accuracy.

Q1. Explaining the AccMP–AccMA Gap in Proprietary Models

To investigate the large AccMP–AccMA gap in P6, we conducted an additional rejection-behavior analysis on the same 24,314 instances across 11 species. The table below reports match-present accuracy (AccMP), false rejection rate (P(E∣MP)P(E \mid \mathrm{MP})), accuracy conditional on selecting an image candidate (A–D) in match-present instances (Accaccept\mathrm{Acc}_{\mathrm{accept}}), and match-absent accuracy (AccMA). Here, E denotes None of the above, and AccMA is equivalent to (P(E∣MA)P(E \mid \mathrm{MA})).

Table: Open-set rejection behavior on P6 (%).

ModelAccMPP(E∣MP)P(E \mid \mathrm{MP})Accaccept\mathrm{Acc}_{\mathrm{accept}}AccMA
Gemini-3.1-Pro4.8987.5244.1448.08
GPT-5.44.2188.8338.6047.93
Claude-Opus-4.65.5487.6147.8948.14
Qwen3.5-122B43.8613.4651.3814.72

The three proprietary models select None of the above on 87.52–88.83% of match-present instances, despite a valid candidate being available. Their rejection rates are considerably lower on match-absent instances (47.93–48.14%), where rejection is the correct decision. This high false rejection rate accounts for most of their match-present errors and explains their low AccMP of 4.21–5.54%.

When selecting an image candidate (A–D) on match-present instances, these models achieve conditional accuracy of 38.60–47.89%, substantially higher than their overall AccMP. In contrast, Qwen3.5-122B exhibits a much lower false rejection rate of 13.46%, with 43.86% AccMP. We have added this analysis to Appendix G.2 and updated the Decision Reliability discussion in the main text.

Q2. Routing Frequency and End-to-End Latency

We evaluated routing frequency and end-to-end latency using the same CLIP/SigLIP retrievers and Qwen3.5-122B verifier as in our retrieval-derived experiments, with the validation-selected routing threshold of ρ=0.2\rho = 0.2. The table below reports the proportion of test queries invoking MLLM verification (Routed Fraction), average latency across all queries (Avg. Latency), and latency among routed queries (Routed Latency).

Table: Routing frequency and end-to-end latency.

RetrieverSettingRouted FractionAvg. Latency (s)Routed Latency (s)
CLIPClosed-set21.19%8.2638.89
SigLIPClosed-set21.16%9.0042.37
CLIPOpen-set21.19%9.5845.10
SigLIPOpen-set21.16%9.5144.79

CLIP and SigLIP route 21.19% and 21.16% of test queries to the MLLM, respectively, leaving approximately 79% handled by retrieval alone. The average per-query latency is 8.26–9.00 s in the closed-set setting and 9.51–9.58 s in the open-set setting. These results show that selective routing reduces the average latency by limiting MLLM verification to approximately one fifth of queries. We have added this analysis to Appendix D.6.

Q3. Comparison with Existing ReID Benchmarks

Thank you for the suggestion. We have added a comparison table in Appendix B (Table 5), covering eight representative person and animal ReID benchmarks alongside ARK. The table compares their evaluation targets, input evidence, decision outputs, and evaluation focus, as shown below.

Table 5. Comparison of representative person and animal ReID benchmarks with ARK.

BenchmarkDomainEvaluation TargetInput EvidenceDecision OutputEvaluation Focus
Market-1501 (2015)PersonGallery retrievalImages; multi-queryRanked imagesCross-camera matching
MARS (2016)PersonVideo retrievalVideo trackletsRanked trackletsMulti-frame matching
DeepChange (2023)PersonLong-term ReIDImagesRanked imagesClothing and temporal changes
FG-PersonX (2026)PersonFine-grained ReIDSynthetic imagesRanked imagesSimilar clothing; controlled visual variation
ATRW (2020)AnimalTiger ReIDImages; pose annotationsRanked imagesPose variation; detection and pose estimation
SeaTurtleID2022 (2024)AnimalLong-term ReIDTimestamped imagesIdentity predictionTime-aware closed/open-set splits
WildlifeReID-10k (2025)AnimalClosed/open-set identificationImagesKnown/unknown identityTime- and similarity-aware splits
MetaWild (2025)AnimalMetadata-assisted ReIDImages; environmental metadataRanked imagesContribution of contextual metadata
ARK (Ours)AnimalMLLM identity decisionsImages; multi-view; context; promptsCandidate selection/rejection; same/differentControlled perception, evidence integration, association and decision reliability

The comparison highlights ARK's focus on controlled evaluation of candidate-level identity decisions made by general-purpose MLLMs. Its seven protocols enable capability-level analysis of identity perception, evidence integration, multi-match association, candidate-set rejection, and sensitivity to misleading suggestions. Further discussion and references are provided in Appendix B.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We thank reviewers u4pu, vn6N, and Jb73, as well as the Area Chairs, for their feedback. The reviews highlighted the value of evaluating MLLM identity verification beyond retrieval ranking and ARK's controlled diagnostic design. During the discussion period, we conducted additional experiments and diagnostic analyses, clarified ARK's positioning relative to existing benchmarks and retrieval-MLLM pipelines, and revised the manuscript accordingly.

1. Benchmark Positioning, Scope, and Reproducibility (u4pu W1-W3, Q3; Jb73 W2)

We added a comparison with eight existing person and animal ReID benchmarks (Appendix B, Table 5) and expanded the discussion of LamRA [1] and Neural Catalog [2]. ARK builds on existing ReID datasets but evaluates a different stage: how general-purpose MLLMs verify retrieved identities, combine evidence, and decide when to reject a candidate set. Across 17 species and seven protocols, it provides a common evaluation of these capabilities instead of measuring embedding quality or gallery ranking alone. The anonymized repository provides benchmark annotations, source mappings, and code for reconstruction (Appendix H).

We also clarified the purpose of Section 4.4 and Appendix D. The retrieval experiments test MLLM verification on actual CLIP and SigLIP shortlists. Their modest closed-set gains, larger open-set improvements, and substantial inference costs help establish where verification is useful and where its practical limitations remain.

2. Additional Experiments and Diagnostic Analyses Addressing Reviewer Questions

Experiment / AnalysisReviewer / ConcernEvaluation and Key Outcome
P6 open-set rejectionu4pu (Q1); Jb73 (W1)Proprietary models incorrectly select "None of the above" on 87.52%–88.83% of match-present instances, explaining much of their low AccMP (Appendix G.2, Table 16).
P7 paired prompt analysisJb73 (W1)Misleading prompts change 67.51% and 69.83% of all pairs from correct to incorrect for Qwen3.5-122B and Claude-Opus-4.6, respectively (Appendix G.4, Table 18).
Routing frequency and latencyu4pu (Q2); vn6N (Q3); Jb73 (W2)At =0.2, 21.19% (CLIP) and 21.16% (SigLIP) of queries invoke the verifier. Average latency is 8.26–9.58 s across closed/open-set evaluations; routing more queries does not consistently improve Rank-1 (Appendix D.6, Table 11, Fig. 6).
Visual re-ranking baselinesvn6N (W2, Q2)Compared KR, ECN, GCR, and Pose2ID-NFC on identical top-5 shortlists. Routed-MLLM achieves 24.15% / 25.50% Rank-1 with CLIP / SigLIP, outperforming all four baselines (best: 22.85% / 25.12%; Appendix D.7, Table 12).
Cross-verifier evaluationJb73 (W2)Across four verifiers, selective routing consistently beats Always-MLLM, improving closed-set Rank-1 by 0.09–1.23 points over Retrieval-only and open-set BAcc by 2.87–4.37 points over Retrieval+Threshold (Appendix D.8, Table 13).
P3 majority-vote tiesJb73 (Q1)No-majority cases occur in 12.4%–18.6% of three-view instances. Joint inference still outperforms random tie-breaking by 0.57–1.61 points (Appendix G.7, Table 19).

3. Clarifications and Manuscript Revisions (vn6N W3, Q4; Jb73 Q2, Q3)

We incorporated the P6 over-rejection and P7 paired-transition findings into the main Results discussion (Section 4.2) and clarified the uncertainty-based routing rule in Section 4.4.

For P4, the existing open-weight Qwen3.5 controls show that full context improves accuracy by 5.46–9.10 points over image-only input, while metadata-only performance remains close to chance (Section 4.3, Fig. 3c). We also discussed the possible influence of pretraining exposure (Appendix A.4).

For output parsing, the P1 first-response audit (Appendix G.1, Table 15) separates initial parsing accuracy from the effects of retry-based recovery, making cross-model comparisons easier to interpret.

[1]. Liu et al., LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

[2]. Khan et al., Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.