acceptodds
Under review as a conference paper at ICLR 2027

IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models

Abstract

Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical applications of VLMs, e.g., where recognizing familiar people and objects is crucial for effective visual understanding. We attribute this limitation in part to the VLM's general-purpose visual representation, which is not explicitly trained to distinguish visually similar instances. To overcome this challenge, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide highly instance-discriminative features complementary to those of the VLM's general-purpose encoder. Through a lightweight two-stage training process, IIR-VLM aligns the expert encoder with the base model, learns to identify new instances in-context, and leverages this knowledge for instance-aware visual understanding. Experiments confirm the benefit of instance-discriminative visual pretraining, particularly for visually similar instances. On IIR-Bench, our challenging new benchmark spanning varying difficulty levels and diverse categories including persons, faces, pets, and general objects, IIR-VLM substantially outperforms strong VLM baselines and prior specialized methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, Instance-level Recognition, In-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sq6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies instance-level recognition in vision-language models, where a query image must be matched to a particular person, face, pet, or object rather than merely assigned a category. IIR-VLM adds an instance-discriminative visual expert to a Qwen2.5-VL-3B model and injects the expert's features into the original visual representation. A two-stage training procedure supports gallery matching and identity-aware captioning. The paper also introduces IIR-Bench, which includes visually similar distractors. On this benchmark, average matching accuracy improves from 90.3% with task-specific fine-tuning without the expert to 92.4% with IIR-VLM.

Critical strengths

The paper addresses a meaningful limitation of current VLMs. Recognizing that an image contains a dog or a person is not the same as recognizing a specific dog or person across different images, and the latter ability is relevant to personalized visual applications.

The proposed framework is relatively simple and the controlled comparisons are informative. In particular, Table 1 shows a clear improvement over task-specific fine-tuning without the expert (90.3% to 92.4%). Table 3 also suggests that specialized instance features are more useful than simply adding a general-purpose visual encoder, although the face expert uses ArcFace while the object/person expert is initialized from DINOv2, and the reported Stage 1 comparison covers three rather than four categories.

The evaluation covers several instance categories and includes results on MM-ID in addition to IIR-Bench. The paper therefore provides useful evidence that instance-discriminative representations can improve gallery matching in a VLM-based system.

Critical weaknesses
  1. The benefit of integrating the expert into the VLM is not sufficiently isolated from simpler expert-based matching.

    The main contribution is not merely using an instance-recognition expert, but incorporating that expert into a general-purpose VLM. However, Eq. (2) already defines direct similarity matching using expert representations, and the experiments do not report an expert-only nearest-neighbor baseline using the same encoder. They also do not compare with a modular approach in which the expert first retrieves the matching identity and that result is then provided to a standard VLM for description or question answering. These comparisons are important because the observed improvement over task-specific VLM fine-tuning is 2.1 percentage points on IIR-Bench (90.3% to 92.4%). This is a meaningful improvement, but it does not by itself establish that internal feature fusion offers an advantage over a simpler use of the same expert.

  2. The claimed identity-aware language capability is not directly evaluated.

    The second training stage is intended to support identity-aware captions, which is a key motivation for integrating the expert with a language model instead of using a standalone recognition system. However, the main quantitative results primarily evaluate which gallery item is selected. There is little systematic evidence about the correctness of identity-aware captions, identity references in generated text, or question answering that requires preserving individual identity. As a result, the paper convincingly demonstrates an improvement in instance matching, but the broader visual-language benefit of the proposed integration remains less clear. A direct comparison on identity-aware description and question-answering tasks would help support this central claim.

Overall, the problem is meaningful and the controlled matching gains are encouraging. However, the two main claims—whether internal fusion is preferable to expert-based retrieval and whether it improves identity-aware language understanding—are not directly validated by the current experiments. These missing comparisons are my main reasons for a slightly below-acceptance assessment.

Questions for authors
  1. How does IIR-VLM compare with direct nearest-neighbor matching using the same expert, and with an expert-retrieval-plus-standard-VLM pipeline, in both accuracy and computational cost?
  2. Could the authors directly evaluate identity-aware captions or identity-dependent question answering, rather than relying mainly on gallery-matching accuracy?
  3. Could the authors report the task-fine-tuned baseline on MM-ID and evaluate larger galleries or cases where no correct matching identity is present?
2 comments

Thank you for the thoughtful review and for finding the problem meaningful and the controlled comparisons informative. We address your concerns below.

1. Expert features and standalone recognition

The expert can perform instance matching directly without a VLM. We compare the expert alone with Vanilla FT and IIR-VLM after Stage 1.

MethodObjectPersonFace
Expert alone89.090.897.6
Vanilla FT, Stage 191.887.491.4
IIR-VLM, Stage 194.490.492.4

On objects, IIR-VLM (94.4) exceeds both the expert alone (89.0) and Vanilla FT (91.8), so the two encoders are complementary and their combination outperforms either one alone. On faces, the expert alone is the strongest matcher. For face recognition tasks, calling ArcFace directly therefore remains a reasonable choice. However, our aim is not to replace dedicated recognizers but to improve the VLM’s own visual representation. Prior work such as Cambrian-1 shows that the visual priors of an encoder affect VLM performance, and our results identify instance discrimination as one such prior that VLM encoders lack, distinct from contrastive learning and SSL. The most relevant test is therefore the comparison with identically trained Vanilla FT, where supplying this prior robustly improves matching in all four categories.

A retrieve-then-caption pipeline takes its identity from the expert, so the expert-only row also gives its matching accuracy. On general objects, IIR-VLM therefore outperforms such a pipeline by 5.4 points, which makes fusion the stronger practical choice. In addition, such a pipeline must also decide which images or frames to send to the recognizer and incorporate the results into its answer, and small VLMs such as our 3B model typically need additional training to call tools reliably. IIR-VLM instead places identity information directly in the visual representation of every image or frame it processes, simplifying the process with a unified architecture.

2. Caption quality and identity-aware visual understanding

Stage 2 trains the model to describe a scene while identifying the relevant instance, so caption quality is an important part of its evaluation. We therefore compare IIR-VLM with the base model and Vanilla FT using CLIP-Image, and also report ROUGE-L, CIDEr and METEOR for objects with reference captions.

ModelObjectPersonFaceAvgMatching accuracy (%)
Base (zero-shot)20.0123.9419.1121.0252.7
Vanilla FT21.0726.7020.7222.8390.3
IIR-VLM20.9626.0820.8122.6292.4
ModelROUGE-LCIDErMETEOR
Vanilla FT43.870.721.0
IIR-VLM43.871.021.0

IIR-VLM improves CLIP-Image over the base model in every category and remains comparable to Vanilla FT (22.62 versus 22.83), while raising matching accuracy from 90.3% to 92.4%. Because IIR-VLM and Vanilla FT describe scenes in similar ways and differ mainly in the identities they assign, their reference scores on objects are also close. These results show that the expert improves instance recognition without disrupting the VLM’s captioning, which is the purpose of Stage 2.

For clarification, scores in Tables 1 and 4 already measure Stage 2's matching performance during caption generation, since we score the identity labels in captions formatted as “[Person 3] walks into the room.” The new metrics complement this evaluation by measuring the quality of the description itself.

3. Computational cost

A cost comparison with the expert alone would not be meaningful, because the expert only matches identities, whereas we aim to teach a VLM to describe and reason about the scene. Retrieve-then-caption, on the other hand, is a relevant alternative. However, that pipeline still requires the expert’s forward pass and VLM generation, and it additionally fills the VLM’s context with tokens that facilitate the tool call and its result. IIR-VLM adds only one forward pass of the expert encoder per image. The expert’s feature is merged into the original encoder’s visual tokens, so IIR-VLM introduces no new tokens.

Revisions to the paper

We have added the expert-only comparison, seed results, caption metrics and this cost discussion to the paper.

We hope these results address the questions you identified as the main reasons for your assessment, namely whether internal fusion is preferable to expert-based retrieval and whether it supports identity-aware language understanding.

Reviewer_HDixOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates In-Context Instance Recognition (IIR) in Vision-Language Models. The authors propose IIR-VLM, which enhances Qwen2.5-VL-3B by integrating a dedicated instance recognition expert. The expert's global representations are projected and injected into the original visual tokens via attention-weighted summation. The model undergoes a two-stage training scheme: first aligning through gallery matching, followed by identity-aware captioning optimized with separately normalized label and caption losses. The paper also introduces IIR-Bench, a novel benchmark spanning four instance categories (objects, persons, faces, and pets), and evaluates instance-matching performance on MM-ID. On IIR-Bench, IIR-VLM achieves a 92.4% average matching accuracy, outperforming both its counterpart trained without the expert module (90.3%) and the zero-shot base model (52.7%).

Critical strengths
  1. The paper addresses a useful limitation of VLMs, and the same-training comparison provides evidence that expert representations help.

  2. The evaluation covers four instance categories and explicitly separates instruction-tuning training and test identities. Together with the MM-ID result, this makes the study more informative than a demonstration confined to a few personalized concepts.

Critical weaknesses
  1. While Equation (2) defines expert-based similarity matching, the expert's standalone matching performance is conspicuously absent from Tables 1–4. This omission is notable because the core benchmarks primarily evaluate instance matching. In addition, while IIR-VLM achieves a 39.7% improvement over the zero-shot base model, much of this gain stems from task-specific fine-tuning; the controlled gain over Vanilla FT is modest at 2.1%, which does not demonstrate that feature-level injection is superior to a simpler two-stage pipeline: expert-based retrieval followed by caption generation with the base VLM. Therefore, the authors could include: 1) a standalone expert baseline evaluating direct similarity matching performance and 2) a retrieval-then-caption baseline using the exact same expert and inputs.

  2. While Sections 1 and 3.3 emphasize leveraging recognized identities to enhance downstream visual understanding, Tables 1–4 exclusively report instance matching accuracy, leaving captioning quality supported only by qualitative examples in Figure 3. Table 4 indicates that matching accuracy largely persists after Stage 2, but it provides no quantitative results that the generated descriptions are more factual or successfully bind actions and attributes to the correct target identities.

  3. The distinction regarding token weighting should be made explicit relative to PLVM [R1], which similarly explores increased weighting on recognition answer tokens. The authors could also compare with closely related paradigms, such as PLVM [R1] and R2P [R2].

  4. To substantiate the broader understanding claim, the authors are encouraged to evaluate identity-reference correctness, caption factuality, and joint metrics against Vanilla FT and retrieval-then-caption baselines on relevant Q&A and captioning tasks in MM-ID [R3].

[R1] Pham C, Phan H, Doermann D, et al. Personalized large vision-language models[J]. arXiv preprint arXiv:2412.17610, 2024. [R2] Das D, Talon D, Wang Y, et al. Training-free personalization via retrieval and reasoning on fingerprints[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 9683-9692. [R3] Ji Y, Zhang S, Wu J, et al. IDA-VLM: towards movie understanding via id-aware large vision-language model[C]//International Conference on Learning Representations. 2025, 2025: 52639-52652.

Questions for authors
  1. On the exact same IIR-Bench examples, how does Equation (2) with your expert compare with IIR-VLM? How does expert retrieval followed by caption generation compare in terms of identity correctness and caption factuality?

  2. Can you provide quantitative caption results and a Stage 2 comparison between ordinary NTP and Equation (5), with the data and number of updates held fixed?

  3. Does identity disjointness apply to both instruction tuning and all expert training data? Please specify the ArcFace checkpoint and overlap checks, the pet data source, and how animal instance labels are derived from the cited iNaturalist data. The latter primarily provides species-level supervision [R5]. Could you provide the expert and two-stage training configurations?

  4. What are the exact evaluation denominators, averaging rules, and run counts? The stated 500 examples per category imply 0.2-point increments for single-run accuracy, whereas some pet scores use other increments.

  5. Which TWIN checkpoint and prompting or conversion procedure were used for the five-candidate task? TWIN is trained primarily on pairwise yes/no judgments [R4].

[R4] Marsili D, Mehta A, Lin R Y, et al. Same or not? enhancing visual perception in vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 17303-17315. [R5] Van Horn G, Mac Aodha O, Song Y, et al. The inaturalist species classification and detection dataset[C]//2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018: 8769-8778.

2 comments

Thank you for the constructive review and for recognizing that the same-training comparison shows the benefit of expert representations. We address your concerns below.

1. Expert features and standalone recognition

The expert can perform instance matching directly without a VLM. We compare the expert alone with Vanilla FT and IIR-VLM after Stage 1.

MethodObjectPersonFace
Expert alone89.090.897.6
Vanilla FT, Stage 191.887.491.4
IIR-VLM, Stage 194.490.492.4

On objects, IIR-VLM (94.4) exceeds both the expert alone (89.0) and Vanilla FT (91.8), so the two encoders are complementary and their combination outperforms either one alone. On faces, the expert alone is the strongest matcher. For face recognition tasks, calling ArcFace directly therefore remains a reasonable choice. However, our aim is not to replace dedicated recognizers but to improve the VLM’s own visual representation. Prior work such as Cambrian-1 shows that the visual priors of an encoder affect VLM performance, and our results identify instance discrimination as one such prior that VLM encoders lack, distinct from contrastive learning and SSL. The most relevant test is therefore the comparison with identically trained Vanilla FT, where supplying this prior robustly improves matching in all four categories.

A retrieve-then-caption pipeline takes its identity from the expert, so the expert-only row also gives its matching accuracy. On general objects, IIR-VLM therefore outperforms such a pipeline by 5.4 points, which makes fusion the stronger practical choice. In addition, such a pipeline must also decide which images or frames to send to the recognizer and incorporate the results into its answer, and small VLMs such as our 3B model typically need additional training to call tools reliably. IIR-VLM instead places identity information directly in the visual representation of every image or frame it processes, simplifying the process with a unified architecture.

2. Caption quality and identity-aware visual understanding

Stage 2 trains the model to describe a scene while identifying the relevant instance, so caption quality is an important part of its evaluation. We therefore compare IIR-VLM with the base model and Vanilla FT using CLIP-Image, and also report ROUGE-L, CIDEr and METEOR for objects with reference captions.

ModelObjectPersonFaceAvgMatching accuracy (%)
Base (zero-shot)20.0123.9419.1121.0252.7
Vanilla FT21.0726.7020.7222.8390.3
IIR-VLM20.9626.0820.8122.6292.4
ModelROUGE-LCIDErMETEOR
Vanilla FT43.870.721.0
IIR-VLM43.871.021.0

IIR-VLM improves CLIP-Image over the base model in every category and remains comparable to Vanilla FT (22.62 versus 22.83), while raising matching accuracy from 90.3% to 92.4%. Because IIR-VLM and Vanilla FT describe scenes in similar ways and differ mainly in the identities they assign, their reference scores on objects are also close. These results show that the expert improves instance recognition without disrupting the VLM’s captioning, which is the purpose of Stage 2.

For clarification, scores in Tables 1 and 4 already measure Stage 2's matching performance during caption generation, since we score the identity labels in captions formatted as “[Person 3] walks into the room.” The new metrics complement this evaluation by measuring the quality of the description itself.

3. Separated supervision

Standard next-token prediction (NTP) averages the loss over all response tokens, so the identity label receives less weight as captions grow longer. Our loss normalizes label and caption tokens separately, which keeps the label’s weight independent of caption length. We compare the two losses below with the same data and number of updates.

StageObjectPersonFaceAvg
Stage 194.490.492.492.4
Stage 1→2, regular NTP91.487.288.489.0
Stage 1→2, separate loss for labels and captions94.290.293.092.5

With our loss, Stage 2 preserves Stage 1 matching accuracy (92.5 versus 92.4 on average), whereas ordinary NTP lowers it by 3.4 points and in every category. Separate supervision therefore allows the model to learn captioning without losing instance matching.

4. Relation to PLVM and R2P

PLVM encodes reference images with DINOv2, and R2P retrieves candidates with CLIP before reasoning over attributes generated by the VLM. Both therefore rely on general-purpose representations, whereas we show that instance discrimination is a distinct visual prior that improves matching beyond an additional DINOv2 encoder. On supervision, PLVM up-weights the Yes/No tokens of its recognition answers, and our loss follows a similar idea. Eq. 5 computes the loss separately on label and caption tokens, so the label keeps the same weight regardless of caption length and Stage 2 preserves the matching capability learned in Stage 1.

5. Data and identity separation

Instruction tuning and our expert training strictly follow the original datasets’ splits, using only training partitions and evaluating on unseen test instances. We use an internal pet-ID dataset with individual labels and will release its evaluation split. For expert training, we use the species label in iNaturalist for fine-grained supervision, whereas the pet instruction-tuning and evaluation data include individual identity labels. We use a public PyTorch implementation of ArcFace with an SE-ResNet-101-IR backbone trained on MS-Celeb-1M.

6. Evaluation and statistical support

Responses whose instance label could not be parsed were excluded from the denominator. This occurred only in a few pet results of the fine-tuned models, which is why those scores fall between 0.2-point steps. All other evaluations used all 500 examples. Averages on IIR-Bench are unweighted means over the four categories.

To ensure that the gain over Vanilla FT is reliable, we repeated Stage 1 with three seeds. IIR-VLM improves mean accuracy over Vanilla FT by 1.4, 3.1 and 3.6 points on objects, persons and faces, and it also exceeds a variant with an additional DINOv2 encoder by 0.7, 2.0 and 2.9 points.

7. TWIN checkpoint and prompting

We used the Qwen2.5-VL-3B TWIN checkpoint with our default prompt, which gave TWIN higher accuracy than its own prompt. Although TWIN’s authors trained the model on binary judgments, their reasoning setup allowed the model to generalize. The "Multi" task in their released benchmark already evaluates instance identification across multiple images, which supports its use in our setting.

Revisions to the paper

We have added the expert-only comparison, seed results, caption metrics and loss ablation to the paper, and discussed PLVM and R2P in related work. For reproducibility, we specified the identity splits, evaluation setup and TWIN setup described above. We also reported the training configuration as implementation detail.

Reviewer_o53HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IIR-VLM augments Qwen2.5-VL-3B with an instance-discriminative expert encoder whose identity embedding is added to the visual tokens through attention-weighted fusion. Training updates only the projectors and proceeds in two stages, gallery matching and identity-aware captioning, with separately normalized losses for label and caption tokens. IIR-Bench covers objects, persons, faces, and pets with visually similar distractors; MM-ID serves as an additional domain. With identical training, the expert raises average matching accuracy from 90.3% for Vanilla FT to 92.4%.

Critical strengths
  1. The method targets a real perceptual gap. Category-level features need not preserve identity cues, and the expert improves person matching from 85.6% to 90.2%.
  2. The gain grows with difficulty. The expert's advantage rises from 0.8 to 3.1 points as distractors become more similar, which ties the improvement to the motivating problem.
  3. The controls are well chosen. Vanilla FT isolates the expert under identical training, the DINOv2 variant shows that an arbitrary extra encoder is not enough, and freezing the LLM keeps the adaptation lightweight.
Critical weaknesses
  1. Direct expert retrieval is never reported. Eq. 2 already matches identities with expert features, yet neither its accuracy on IIR-Bench nor a retrieve-then-caption baseline is given. Without them, the value of fusion is unclear, especially since the expert adds 2.1 points while in-domain fine-tuning adds 37.6.
  2. The understanding claim is untested. The paper claims gains in instance-aware understanding, but only matching accuracy is measured, and the separate label–caption loss is never ablated. The abstract also overstates the comparison with strong VLMs, since Gemini 3.8 Flash scores higher on average, 94.2% versus 92.4%.
  3. The statistical support is thin. Several gains are below two points on 500 queries per category, with no seeds or confidence intervals, and the MM-ID results appear to rest on about 67 items, so the lead over IDA-VLM amounts to five items.
Questions for authors
  1. On the same galleries, how do direct nearest-neighbor matching with the identical expert, as in Eq. 2, and expert retrieval followed by VLM captioning compare with feature fusion?
  2. What quantitative results establish the benefits of Stage 2 for identity-reference correctness and caption factuality? How does the separate loss in Eq. 5 compare with standard next-token supervision?
  3. How many items does each MM-ID category contain? The reported percentages are consistent with 28 face, 12 animal, 17 building, and 10 vehicle items. Please add confidence intervals and a Vanilla FT result on MM-ID, and seeds or confidence intervals for the IIR-Bench gains.
  4. Which encoder defines the distractor-mining threshold τ in Sec. 3.3? If it is DINOv2 or the VLM's own encoder, benchmark difficulty is defined relative to the representation the expert complements. Do the results hold when distractors are mined with the expert's features?
  5. Please specify the expert-training and evaluation identity splits, the pet-data source, and the ArcFace checkpoint, so that identity separation can also be verified for expert pretraining. This is a request to establish the protocol, not an allegation of leakage.
2 comments

Thank you for the careful review and for noting that the gain grows with distractor similarity and that controls are well chosen. We address your concerns below.

1. Expert features and standalone recognition

The expert can perform instance matching directly without a VLM. We compare the expert alone with Vanilla FT and IIR-VLM after Stage 1.

MethodObjectPersonFace
Expert alone89.090.897.6
Vanilla FT, Stage 191.887.491.4
IIR-VLM, Stage 194.490.492.4

On objects, IIR-VLM (94.4) exceeds both the expert alone (89.0) and Vanilla FT (91.8), so the two encoders are complementary and their combination outperforms either one alone. On faces, the expert alone is the strongest matcher. For face recognition tasks, calling ArcFace directly therefore remains a reasonable choice. However, our aim is not to replace dedicated recognizers but to improve the VLM’s own visual representation. Prior work such as Cambrian-1 shows that the visual priors of an encoder affect VLM performance, and our results identify instance discrimination as one such prior that VLM encoders lack, distinct from contrastive learning and SSL. The most relevant test is therefore the comparison with identically trained Vanilla FT, where supplying this prior robustly improves matching in all four categories.

A retrieve-then-caption pipeline takes its identity from the expert, so the expert-only row also gives its matching accuracy. On general objects, IIR-VLM therefore outperforms such a pipeline by 5.4 points, which makes fusion the stronger practical choice. In addition, such a pipeline must also decide which images or frames to send to the recognizer and incorporate the results into its answer, and small VLMs such as our 3B model typically need additional training to call tools reliably. IIR-VLM instead places identity information directly in the visual representation of every image or frame it processes, simplifying the process with a unified architecture.

2. Statistical support

We repeated Stage 1 with three seeds for Vanilla FT, a variant with an additional DINOv2 encoder, and IIR-VLM, and report the mean and standard deviation below.

MethodObjectPersonFaceAvg
Vanilla FT92.27 ± 1.3086.87 ± 0.9290.40 ± 0.8789.84 ± 0.79
DINOv293.00 ± 0.6987.93 ± 1.3391.13 ± 0.7090.69 ± 0.56
IIR-VLM93.67 ± 1.1089.93 ± 0.6494.00 ± 0.8792.53 ± 0.12

IIR-VLM improves mean accuracy over Vanilla FT by 1.4, 3.1 and 3.6 points on objects, persons and faces and also exceeds the DINOv2 variant in every category. For objects and persons, where our expert is initialized from DINOv2, this comparison attributes the gain to instance-discriminative training rather than to the additional encoder.

3. Caption quality and identity-aware visual understanding

Stage 2 trains the model to describe a scene while identifying the relevant instance, so caption quality is an important part of its evaluation. We therefore compare IIR-VLM with the base model and Vanilla FT using CLIP-Image, and also report ROUGE-L, CIDEr and METEOR for objects with reference captions.

ModelObjectPersonFaceAvgMatching accuracy (%)
Base (zero-shot)20.0123.9419.1121.0252.7
Vanilla FT21.0726.7020.7222.8390.3
IIR-VLM20.9626.0820.8122.6292.4
ModelROUGE-LCIDErMETEOR
Vanilla FT43.870.721.0
IIR-VLM43.871.021.0

IIR-VLM improves CLIP-Image over the base model in every category and remains comparable to Vanilla FT (22.62 versus 22.83), while raising matching accuracy from 90.3% to 92.4%. Because IIR-VLM and Vanilla FT describe scenes in similar ways and differ mainly in the identities they assign, their reference scores on objects are also close. These results show that the expert improves instance recognition without disrupting the VLM’s captioning, which is the purpose of Stage 2.

For clarification, scores in Tables 1 and 4 already measure Stage 2's matching performance during caption generation, since we score the identity labels in captions formatted as “[Person 3] walks into the room.” The new metrics complement this evaluation by measuring the quality of the description itself.

4. Separated supervision

Standard next-token prediction (NTP) averages the loss over all response tokens, so the identity label receives less weight as captions grow longer. Our loss normalizes label and caption tokens separately, which keeps the label’s weight independent of caption length. We compare the two losses below.

StageObjectPersonFaceAvg
Stage 194.490.492.492.4
Stage 1→2, regular NTP91.487.288.489.0
Stage 1→2, separate loss for labels and captions94.290.293.092.5

With our loss, Stage 2 preserves Stage 1 matching accuracy (92.5 versus 92.4 on average), whereas ordinary NTP lowers it by 3.4 points and in every category. Separate supervision therefore allows the model to learn captioning without losing instance matching.

5. Distractor mining

We deliberately mine distractors with a general DINOv2 backbone rather than the expert. IIR-Bench is designed around instances that general representations find similar, which is the weakness of the VLM's existing encoder. This is what instance-discriminative features are meant to address. The expert’s advantage grows from 0.8 to 3.1 points as τ increases (Fig. 4), which confirms that it helps most where general representations struggle.

6. Data and implementation detail

Instruction tuning and our expert training strictly follow the original datasets’ splits, using only training partitions and evaluating on unseen test instances. We use an internal pet-ID dataset with individual labels and will release its evaluation split. We use a public PyTorch implementation of ArcFace with an SE-ResNet-101-IR backbone trained on MS-Celeb-1M.

Revisions to the paper

We have added the expert-only comparison, seed results, caption metrics and loss ablation to the paper. We also specified the mining encoder and identity splits described above. Finally, we revised the abstract to make sure its claims are accurate.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author summary of rebuttal

Dear PC, AC, and reviewers,

We thank the reviewers for acknowledging the meaningful VLM limitation we address (o53H, HDix, sq6H), the informative controlled comparisons (o53H, HDix, sq6H), the expert advantage that grows with difficulty (o53H), and the broadness of our evaluation (HDix, sq6H).

Here we provide a summary of our response, particularly to commonly raised questions. The reviewers asked about expert-only and retrieve-then-caption baselines, caption quality and the reliability of the gains. Our paper asks whether instance-recognition features improve VLM perception, and the rebuttal adds evidence with each point.

  • Comparison with the individual expert. On general objects, IIR-VLM (94.4) outperforms both the expert alone (89.0) and Vanilla FT (91.8), so the two encoders are complementary. Because a retrieve-then-caption pipeline inherits the expert’s accuracy, IIR-VLM also exceeds it by 5.4 points while avoiding tool calls, which small VLMs typically cannot make reliably without additional training. On faces, the expert alone is the strongest matcher, and calling it directly remains a reasonable choice for face recognition tasks. Our focus, however, is improving the VLM’s own visual representation. We show that instance discrimination is a visual prior that VLM encoders lack, and supplying it improves matching over identically trained Vanilla FT in all four categories.
  • Consistent gains across seeds. Over three seeds of Stage 1 training, IIR-VLM improves mean accuracy over identically trained Vanilla FT by 1.4, 3.1 and 3.6 points on objects, persons and faces. It also exceeds a variant with an additional DINOv2 encoder by 0.7, 2.0 and 2.9 points, which attributes the gain to instance-discriminative training rather than to an extra encoder.
  • Retained caption quality. Separate label–caption supervision preserves Stage 1 matching during caption training (92.5 vs. 92.4), whereas standard next-token prediction lowers it to 89.0. IIR-VLM improves CLIP-Image scores over the base model in every category and remains comparable to Vanilla FT, while raising instance matching accuracy. The expert therefore improves instance recognition without disrupting the VLM’s captioning.

More details are included in our individual response to the reviewers. We have added these results and the protocol details to the paper, with changes highlighted in blue. We thank the reviewers for contributing to strengthening the arguments of the paper.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.