- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies instance-level recognition in vision-language models, where a query image must be matched to a particular person, face, pet, or object rather than merely assigned a category. IIR-VLM adds an instance-discriminative visual expert to a Qwen2.5-VL-3B model and injects the expert's features into the original visual representation. A two-stage training procedure supports gallery matching and identity-aware captioning. The paper also introduces IIR-Bench, which includes visually similar distractors. On this benchmark, average matching accuracy improves from 90.3% with task-specific fine-tuning without the expert to 92.4% with IIR-VLM.
Critical strengths
The paper addresses a meaningful limitation of current VLMs. Recognizing that an image contains a dog or a person is not the same as recognizing a specific dog or person across different images, and the latter ability is relevant to personalized visual applications.
The proposed framework is relatively simple and the controlled comparisons are informative. In particular, Table 1 shows a clear improvement over task-specific fine-tuning without the expert (90.3% to 92.4%). Table 3 also suggests that specialized instance features are more useful than simply adding a general-purpose visual encoder, although the face expert uses ArcFace while the object/person expert is initialized from DINOv2, and the reported Stage 1 comparison covers three rather than four categories.
The evaluation covers several instance categories and includes results on MM-ID in addition to IIR-Bench. The paper therefore provides useful evidence that instance-discriminative representations can improve gallery matching in a VLM-based system.
Critical weaknesses
-
The benefit of integrating the expert into the VLM is not sufficiently isolated from simpler expert-based matching.
The main contribution is not merely using an instance-recognition expert, but incorporating that expert into a general-purpose VLM. However, Eq. (2) already defines direct similarity matching using expert representations, and the experiments do not report an expert-only nearest-neighbor baseline using the same encoder. They also do not compare with a modular approach in which the expert first retrieves the matching identity and that result is then provided to a standard VLM for description or question answering. These comparisons are important because the observed improvement over task-specific VLM fine-tuning is 2.1 percentage points on IIR-Bench (90.3% to 92.4%). This is a meaningful improvement, but it does not by itself establish that internal feature fusion offers an advantage over a simpler use of the same expert.
-
The claimed identity-aware language capability is not directly evaluated.
The second training stage is intended to support identity-aware captions, which is a key motivation for integrating the expert with a language model instead of using a standalone recognition system. However, the main quantitative results primarily evaluate which gallery item is selected. There is little systematic evidence about the correctness of identity-aware captions, identity references in generated text, or question answering that requires preserving individual identity. As a result, the paper convincingly demonstrates an improvement in instance matching, but the broader visual-language benefit of the proposed integration remains less clear. A direct comparison on identity-aware description and question-answering tasks would help support this central claim.
Overall, the problem is meaningful and the controlled matching gains are encouraging. However, the two main claims—whether internal fusion is preferable to expert-based retrieval and whether it improves identity-aware language understanding—are not directly validated by the current experiments. These missing comparisons are my main reasons for a slightly below-acceptance assessment.
Questions for authors
- How does IIR-VLM compare with direct nearest-neighbor matching using the same expert, and with an expert-retrieval-plus-standard-VLM pipeline, in both accuracy and computational cost?
- Could the authors directly evaluate identity-aware captions or identity-dependent question answering, rather than relying mainly on gallery-matching accuracy?
- Could the authors report the task-fine-tuned baseline on MM-ID and evaluate larger galleries or cases where no correct matching identity is present?
2 comments
2. Caption quality and identity-aware visual understanding
Stage 2 trains the model to describe a scene while identifying the relevant instance, so caption quality is an important part of its evaluation. We therefore compare IIR-VLM with the base model and Vanilla FT using CLIP-Image, and also report ROUGE-L, CIDEr and METEOR for objects with reference captions.
| Model | Object | Person | Face | Avg | Matching accuracy (%) |
|---|---|---|---|---|---|
| Base (zero-shot) | 20.01 | 23.94 | 19.11 | 21.02 | 52.7 |
| Vanilla FT | 21.07 | 26.70 | 20.72 | 22.83 | 90.3 |
| IIR-VLM | 20.96 | 26.08 | 20.81 | 22.62 | 92.4 |
| Model | ROUGE-L | CIDEr | METEOR |
|---|---|---|---|
| Vanilla FT | 43.8 | 70.7 | 21.0 |
| IIR-VLM | 43.8 | 71.0 | 21.0 |
IIR-VLM improves CLIP-Image over the base model in every category and remains comparable to Vanilla FT (22.62 versus 22.83), while raising matching accuracy from 90.3% to 92.4%. Because IIR-VLM and Vanilla FT describe scenes in similar ways and differ mainly in the identities they assign, their reference scores on objects are also close. These results show that the expert improves instance recognition without disrupting the VLM’s captioning, which is the purpose of Stage 2.
For clarification, scores in Tables 1 and 4 already measure Stage 2's matching performance during caption generation, since we score the identity labels in captions formatted as “[Person 3] walks into the room.” The new metrics complement this evaluation by measuring the quality of the description itself.
3. Computational cost
A cost comparison with the expert alone would not be meaningful, because the expert only matches identities, whereas we aim to teach a VLM to describe and reason about the scene. Retrieve-then-caption, on the other hand, is a relevant alternative. However, that pipeline still requires the expert’s forward pass and VLM generation, and it additionally fills the VLM’s context with tokens that facilitate the tool call and its result. IIR-VLM adds only one forward pass of the expert encoder per image. The expert’s feature is merged into the original encoder’s visual tokens, so IIR-VLM introduces no new tokens.
Revisions to the paper
We have added the expert-only comparison, seed results, caption metrics and this cost discussion to the paper.
We hope these results address the questions you identified as the main reasons for your assessment, namely whether internal fusion is preferable to expert-based retrieval and whether it supports identity-aware language understanding.
Thank you for the thoughtful review and for finding the problem meaningful and the controlled comparisons informative. We address your concerns below.
1. Expert features and standalone recognition
The expert can perform instance matching directly without a VLM. We compare the expert alone with Vanilla FT and IIR-VLM after Stage 1.
On objects, IIR-VLM (94.4) exceeds both the expert alone (89.0) and Vanilla FT (91.8), so the two encoders are complementary and their combination outperforms either one alone. On faces, the expert alone is the strongest matcher. For face recognition tasks, calling ArcFace directly therefore remains a reasonable choice. However, our aim is not to replace dedicated recognizers but to improve the VLM’s own visual representation. Prior work such as Cambrian-1 shows that the visual priors of an encoder affect VLM performance, and our results identify instance discrimination as one such prior that VLM encoders lack, distinct from contrastive learning and SSL. The most relevant test is therefore the comparison with identically trained Vanilla FT, where supplying this prior robustly improves matching in all four categories.
A retrieve-then-caption pipeline takes its identity from the expert, so the expert-only row also gives its matching accuracy. On general objects, IIR-VLM therefore outperforms such a pipeline by 5.4 points, which makes fusion the stronger practical choice. In addition, such a pipeline must also decide which images or frames to send to the recognizer and incorporate the results into its answer, and small VLMs such as our 3B model typically need additional training to call tools reliably. IIR-VLM instead places identity information directly in the visual representation of every image or frame it processes, simplifying the process with a unified architecture.