- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper formulates Reasoning ReID, a post-retrieval MLLM verification stage, and introduces ARK, an Animal ReID benchmark of 187,865 evaluation instances across 17 species and seven diagnostic protocols organized into three capability families. An evaluation of 21 MLLMs alongside a human reference shows that models can benefit from complementary evidence, while their reliability remains uneven under demanding verification conditions.
Critical strengths
- Beyond single-match accuracy, ARK evaluates multi-match identity association, open-set rejection, and resistance to misleading suggestions, which retrieval metrics alone do not capture. For example, P5 shows distinct precision–recall profiles: Qwen3.5-122B is balanced (71.38% precision, 67.71% recall), whereas Gemma3-27B over-selects (73.70% recall, 49.77% precision).
- Controlled diagnostic design. ARK holds candidate sets fixed across corruption, multi-view, and context conditions, and image pairs fixed across prompt conditions. These controls help distinguish the effects of the evaluated conditions from changes in candidate composition.
Critical weaknesses
- Decision Reliability needs further diagnosis. Proprietary models achieve 43.64–53.49% accuracy on P1 but only 4.21–5.54% AccMP on P6, and 47.93–48.14% AccMA. Because the parsing audit covers only P1, the source of this drop remains unresolved. In P7, LLaVA-13B’s leading AccM among the models shown accompanies chance-level AccN, so paired Neutral/Misleading predictions are needed to measure how often correct Neutral judgments are preserved under misleading prompts.
- Limited validation of pipeline gains. Routed-MLLM improves closed-set accuracy by around one percentage point, while Always-MLLM underperforms Retrieval-only. Only one verifier is evaluated, requiring approximately 40 seconds per verification call, and the routing frequency and average Routed-MLLM latency are unreported.
Questions for authors
- In the P3 comparison with N = 3 (Fig. 3b), what fraction of instances lack a majority under Vote for Qwen3-VL-8B, Qwen3.5-35B, and Qwen3.5-122B? Does Joint still outperform Vote when ties are resolved uniformly at random among the tied predictions, measured as expected accuracy?
- Key information needed to interpret the main results appears only in the appendix, such as the Routed-MLLM routing rule.
- The presentation is overextended and could be more focused. Extensive procedural descriptions occupy substantial space, while some results remain insufficiently explained, such as unexpected Decision Reliability results.
4 comments
Q1. Majority-Vote Tie-Breaking Sensitivity
Thank you for this suggestion. We examine no-majority cases in P3 () and evaluate uniform random tie-breaking, with expected accuracy computed analytically. The results are summarized below.
| Model | No Majority (%) | Strict Vote (%) | Random-Tie Vote (%) | Joint (%) | Δ (pp) |
|---|---|---|---|---|---|
| Qwen3-VL-8B | 18.6 | 35.08 | 36.63 | 37.20 | +0.57 |
| Qwen3.5-35B | 14.9 | 48.56 | 50.65 | 51.96 | +1.31 |
| Qwen3.5-122B | 12.4 | 55.47 | 57.83 | 59.44 | +1.61 |
No Majority is pooled; accuracies are species-macro averaged. Δ = Joint − Random-Tie Vote.
Joint continues to outperform Random-Tie Vote across all three models, confirming that its advantage is not solely attributable to the treatment of no-majority cases. We have added the detailed results and analysis to Appendix G.7.
Q2. Routing Clarity in the Main Paper
We have revised Section 4.4 to clarify how Routed-MLLM identifies uncertain retrievals. Retrieval uncertainty is measured using the margin between the top-1 and top-2 retrieval scores, normalized to a species-specific percentile based on validation data. Lower percentiles indicate greater retrieval uncertainty, and queries falling below a validation-selected threshold are routed to the MLLM verifier.
Q3. Presentation Focus and Result Interpretation
We have strengthened the main-text discussion by providing clearer explanations of the Decision Reliability findings. For P6, our new rejection-behavior analysis identifies excessive selection of "None of the above" as the primary explanation for the proprietary models' low match-present accuracy (Appendix G.2). For P7, the paired Neutral–Misleading analysis provides a more detailed understanding of model robustness by distinguishing the preservation of correct judgments from accuracy under individual prompting conditions (Appendix G.4). These findings are now incorporated into the main Results discussion, providing a clearer interpretation of the observed performance patterns.
Thank you for the detailed rebuttal.
W2. While the authors extend the evaluation to four verifiers, the closed-set gains are small, and most gains being marginal. Moreover, the gains show no consistent relationship with model size and appear somewhat arbitrary, making it unclear when or why MLLM verification helps. Considering the substantial inference latency introduced by the verifiers, the practical significance of these improvements are limited. Although the open-set results are much better, my concern about the practical value of MLLM verification and its connection to the benchmark's evaluated capabilities remains.
Therefore, I will maintain my original score.
Thank you for the follow-up. We would like to further clarify the patterns in the cross-verifier results and the contribution of ARK.
Verifier Performance. Appendix D.8 shows consistent patterns across both retrievers. All four verifiers achieve positive gains with selective routing, and their performance rankings are identical between CLIP and SigLIP within each evaluation setting. Within the Qwen3.5 family, scaling from 35B to 122B improves closed-set gains from 0.49–0.55 to 1.02–1.12 percentage points, and open-set gains from 2.87 to 4.36–4.37 points. Across different model families, parameter count alone is an insufficient predictor of performance. For example, despite similar nominal parameter counts, Qwen3-VL-30B substantially outperforms Gemma3-27B on P1 (36.96% vs. 24.83%) and P3 (39.79% vs. 24.25%), as shown in Table 1. Furthermore, Routed-MLLM consistently outperforms Always-MLLM across all four verifiers and both retrievers, highlighting the importance of selective verification.
Benchmark Contribution and Practical Value. ARK aims to systematically characterize the capabilities and limitations of general-purpose MLLMs as identity verifiers. Through seven controlled protocols and an evaluation of 21 MLLMs, ARK investigates visual identity perception, evidence integration and association, and decision reliability. Section 4.4 and Appendix D.8 further examine these verifiers in retrieval-based pipelines. The modest closed-set gains, stronger open-set improvements, and substantial inference costs reveal both the potential and current limitations of MLLM verification. These findings provide insights into verifier behavior and highlight challenges for future MLLM-based ReID research.
Responses to Reviewer Jb73
W1. Decision Reliability: Open-Set Rejection and Resistance to Misleading Suggestions
To investigate the unusually low P6 match-present accuracy, we add a rejection-behavior analysis of three proprietary MLLMs and Qwen3.5-122B across 24,314 instances (Appendix G.2). The proprietary models incorrectly select "None of the above" on 87.52–88.83% of match-present instances, despite a valid match being available. These are parseable but incorrect decisions, demonstrating that excessive rejection, rather than output-format failures, is the primary explanation for their low AccMP (4.21–5.54%). The three proprietary models also achieve 99.96–99.99% first-response parsing validity in our P1 audit (Appendix G.1).
We have added a paired analysis of LLaVA-13B, Qwen3.5-122B, and Claude-Opus-4.6 using identical image pairs under Neutral and Misleading prompts (Appendix G.4). We report four correctness transitions and Correctness Preservation, defined as the proportion of neutral-correct judgments that remain correct under misleading prompts, macro-averaged across 11 species. The results show that strong performance under Neutral prompts does not guarantee resistance to misleading suggestions. Qwen3.5-122B and Claude-Opus-4.6 achieve 72.66% and 73.52% AccN, but 67.51% and 69.83% of all pairs transition from correct to incorrect, with correctness preservation rates of only 6.85% and 4.85%, respectively. In contrast, LLaVA-13B achieves relatively high AccM (44.84%) and preservation (74.55%). However, its AccN is already near chance (49.49%), and 42.57% of pairs remain incorrect under both prompts. Its relatively high AccM and preservation therefore do not establish reliable identity verification.
W2. Pipeline Gains, Verifier Generality, and Inference Efficiency
Our retrieval-derived evaluation assesses the value of MLLM verification under both closed-set identification and open-set shortlist rejection. With Qwen3.5-122B, Routed-MLLM improves closed-set Rank-1 accuracy over Retrieval-only by 1.02 and 1.12 percentage points for CLIP and SigLIP, respectively, while increasing open-set BAcc over Retrieval+Threshold by 4.36 and 4.37 points (Section 4.4; Appendix D.4). These larger open-set gains highlight the value of MLLM verification for shortlist rejection beyond conventional closed-set ranking.
To further assess verifier generality, we extend the evaluation to four verifiers: Qwen3.5-122B, Qwen3.5-35B, Qwen3-VL-235B, and Claude-Opus-4.6 (Appendix D.8). Routed-MLLM improves closed-set Rank-1 accuracy over Retrieval-only by 0.09–1.23 percentage points and open-set BAcc over Retrieval+Threshold by 2.87–4.37 points. Moreover, Routed-MLLM consistently outperforms Always-MLLM across all four verifiers and both retrievers, supporting selective rather than uniform MLLM verification.
Regarding inference efficiency, an individual verification call with Qwen3.5-122B takes approximately 40 seconds. At the selected threshold, only 21.19% and 21.16% of queries invoke the verifier for CLIP and SigLIP, respectively. This yields average latencies of 8.26 s and 9.00 s under closed-set identification, and 9.58 s and 9.51 s under open-set evaluation (Appendix D.5–D.6). Our routing-threshold analysis further shows that increasing verification frequency generally raises latency without improving closed-set accuracy, supporting selective routing as a more favorable accuracy–latency trade-off than full verification.