acceptodds
Withdrawn from ICLR 2027

Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalizing Vision-Language Models (VLMs) to transform them into daily assistants has emerged as a trending research direction. However, leading companies like OpenAI continue to increase model size and develop complex designs such as the chain of thought (CoT). While large VLMs are proficient in complex multi-modal understanding, their high training costs and limited access via paid APIs restrict direct personalization. Conversely, small VLMs are easily personalized and freely available, but they lack sufficient reasoning capabilities. Inspired by this, we propose a novel collaborative framework named Small-Large Collaboration (SLC) for large VLM personalization, where the small VLM is responsible for generating personalized information, while the large model integrates this personalized information to deliver accurate responses. To effectively incorporate personalized information, we develop a test-time reflection strategy, preventing the potential hallucination of the small VLM. Since SLC only needs to train a meta personalized small VLM for the large VLMs, the overall process is training-efficient. To the best of our knowledge, this is the first training-efficient framework that supports both open-source and closed-source large VLMs, enabling broader real-world personalized applications. We conduct thorough experiments across various benchmarks and large VLMs to demonstrate the effectiveness of the proposed SLC framework. The code and data will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language model, personalization, meta-learning, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hjR5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Small-Large Collaboration for training-efficient VLM personalization. The method trains a small VLM using CLIP-based clustering and meta-concept LoRA adapters. A large VLM verifies personalized concepts through test-time reflection. Experiments on Yo'LLaVA and MC-LLaVA show competitive performance with lower training costs.

Critical strengths
  1. The proposed framework avoids direct fine-tuning of large VLMs and supports both open-source and closed-source models.

  2. The small-large collaboration design is modular and practical. The combination of meta-concept adapters and test-time reflection provides a potentially useful approach to personalized inference.

Critical weaknesses
  1. MyVLM [1] already combines external personalized concept recognition with VLM generation. RAP [2] supports new concept registration without per-concept fine-tuning. Cache-of-Thought [3] explores small-large VLM collaboration. The paper should clarify its technical novelty beyond combining existing components and provide stronger comparisons with these methods.

  2. Section 3.3 uses only 10 meta-concept adapters selected through CLIP similarity. Different identities with similar appearances may select the same adapter. It is unclear how the model distinguishes such identities without explicitly preserving identity-specific reference information. This raises concerns about category-level recognition rather than genuine instance-level personalization [1,4,5].

  3. Inconsistent adapter selection descriptions. Equation (1) selects one adapter using averaged embeddings from all registered concepts. However, Figure 3 suggests independent adapter selection for each concept. These strategies differ significantly in multi-concept scenarios.

  4. Insufficient identity verification in reflection. Section 3.4.2 verifies broad category descriptions and spatial locations rather than unique identities. A visually similar but incorrect object may satisfy both verification questions. Incorrect spatial predictions may also cause true concepts to be removed.

  5. Numerical inconsistencies. Figure 1 claims the highest average accuracy for SLC (0.933), but Table 1 gives MC-LLaVA a higher average of 0.9405. The claimed two-order training-cost reduction is only about 41 times relative to MC-LLaVA. Section 4.2.2 reports SQA accuracy as 0.900, while Table 1 reports 0.925. The GPT-4o results discussed in Section 4.2.2 also differ from the GPT-5.5 results in Table 1.

  6. Section 4.3.1 claims six settings plus two baselines, but Table 2 contains six rows in total.

References:

[1] Alaluf et al. MyVLM: Personalizing VLMs for User-Specific Queries. ECCV, 2024.

[2] Hao et al. RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models. CVPR, 2025.

[3] Wu et al. Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Inference. arXiv:2502.20587, 2025.

[4] Nguyen et al. Yo'LLaVA: Your Personalized Language and Vision Assistant. NeurIPS, 2024.

[5] Pi et al. Personalized Visual Instruction Tuning. arXiv:2410.07113, 2024.

Questions for authors

Q1. How does the small VLM distinguish unseen identities with similar appearances and the same selected adapter? Are reference images or identity-specific embeddings used beyond adapter selection? Please provide fine-grained identity recognition and adapter collision experiments.

Q2. Is adapter selection performed per concept or per scenario? Equation (1) and Figure 3 appear inconsistent. How are multiple concepts from different categories handled during inference?

Q3. What advantages does SLC provide over retrieval-based personalization methods such as RAP? Could the authors compare SLC with direct CLIP retrieval and lightweight external concept detectors under matched settings?

Reviewer_4ggQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An end-to-end "small-large collaborative" framework is proposed, which decouples personalized tasks into detection (performed by a small VLM) and verification/reasoning (performed by a large VLM), thereby overcoming the challenge that closed-source large models cannot be directly fine-tuned.

Critical strengths
  1. Comprehensive comparative evaluations are reported across two public benchmarks, covering identification, VQA, text-only QA, and SQA; the framework demonstrates competitive performance against existing models.

  2. The authors introduce a Self-Questioning dataset designed to target hallucinations, testing whether a model relies on reasoning based on the current image rather than on memorized information.

  3. Decoupling personalization tasks into detection and verification/reasoning addresses the challenge that closed-source large models cannot be directly fine-tuned.

Critical weaknesses
  1. Table 1 compares the proposed method only against Yo'LLaVA, MC-LLaVA, and RAP-LLaVA—all of which are based on LLaVA-1.5-13B. The model should be compared against more recent state-of-the-art (SOTA) methods, and the selection of baselines should be more diverse.

  2. The model's effectiveness is primarily validated through experiments, lacking theoretical analysis.

  3. The paper only compares training FLOPs; it fails to report inference latency, token costs, or wall-clock time.

  4. The SQA dataset is extremely small, consisting of only 80 questions. Was this dataset generated by AI or written by humans? Furthermore, given the use of a VLM as a judge, is there a risk of the judge hallucinating? The study lacks evaluations by human experts.

Questions for authors
  1. When a VLM is used as a "judge," is there a risk that the judge itself might hallucinate?

  2. The SQA dataset is extremely small, consisting of only 80 questions. Was this dataset generated by AI or authored by humans?

  3. How were the 80 questions in the SQA dataset constructed? Given its very small size (80 questions), was it generated by AI or written by humans?

  4. The "reflection" process involves three separate inferences by a large-scale model, which incurs significant token costs. How do the authors justify that this reasoning approach remains competitive in terms of efficiency?

Reviewer_2HrpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Small–Large Collaboration (SLC) for personalizing large vision-language models. A small VLM uses meta-concept LoRA adapters to produce concept detections, which a frozen large VLM checks and uses to answer questions. The method is evaluated on Yo’LLaVA, MC-LLaVA and an additional SQA set, with comparisons of prediction accuracy, training cost and model components.

Critical strengths
  • The large reader stays frozen, and registration requires no further tuning. Table 1 reports substantially lower offline training cost than the selected fine-tuning baselines.
  • With the same LLaVA-1.5-13B reader, SLC improves recognition and VQA over text prompting on both datasets. The reader comparison also shows gains from stronger large VLMs.
  • SQA tests whether the model repeats registration-time attributes when the query image contradicts them. These examples and the component ablations expose failures beyond aggregate recognition accuracy.
Critical weaknesses
  • The paper does not explain how the detector distinguishes identities defined by the reference images when their descriptions are insufficient. Section 3.3.2 pools reference features to select one shared adapter, while Table 6 supplies the detector with the query image and identifier-to-description text. For two visually distinct unseen instances with identical descriptions, swapping their reference-to-ID assignments leaves these inputs unchanged but changes the correct ID. Section 3.1 does not require uniquely identifying descriptions. The method needs either a per-ID reference channel or an explicit restriction to description-distinguishable concepts.

  • Reflection's effect on personal identity is unclear. The verifier checks a text-derived phrase at two locations (Section 3.4.2). Figure 9 includes fine attributes, but another instance matching the same phrase could still receive the wrong personal ID. Table 4 also reports higher Yes recall after reflection, although Appendix 9 says the filter does not revisit false negatives and Table 9 makes final presence answers follow report labels. The paper does not identify the measured stage or whether both ablations reuse the same raw detections.

  • The comparison with training-free personalization methods is incomplete. PeKit [1] combines reference-based instance matching with an unchanged reader and evaluates multiple concepts; R2P [2] combines concept inference with attribute verification. The abstract's first-framework claim needs to distinguish SLC's meta-adapter contribution from these alternatives.

    Table 1 also shows an accuracy trade-off with the same LLaVA reader. Compared with MC-LLaVA, SLC improves Yo’LLaVA VQA but trails in Yo’LLaVA recognition and in weighted recognition/VQA on the MC-LLaVA dataset. A comparison with a relevant training-free method under matched conditions is missing.

  • Section 4.1.2 excludes images associated with training meta-concepts, and Appendix 9 states identity disjointness, but neither gives the remaining identity, scenario and query counts. If representative selection changes the exclusions across Figure 4 library sizes, those results may compare different populations.

  • The routing ablation does not isolate the benefit of matching a scenario to its nearest adapter. Table 2 tests LoRA and prompting, while Figure 4 compares nearest Top-1 with averaging several nearby adapters. Neither compares nearest selection against a fixed or random adapter from the same bank.

My main concern is the missing reference-to-ID information when descriptions do not identify an instance. The training-efficiency claim also needs comparison with relevant training-free approaches.

[1] Seifi et al. Personalization Toolkit (PeKit), TMLR 2026.

[2] Das et al. R2P, ICCV 2025.

Questions for authors
  1. After adapter selection, what per-ID reference information reaches the detector beyond the text description? What changes if two unseen instances have identical descriptions and their reference-to-ID assignments are swapped?
  2. Please identify whether Table 4 reports recall for raw detections, filtered reports or final answers, and whether the ablations reuse the same raw detections. Does reflection reject an incorrect personal ID when the instance still matches the extracted phrase?
  3. Please distinguish the abstract's first-framework claim from PeKit and R2P, and compare SLC with a relevant training-free method using the same reader, registration information and evaluation population. If the efficiency claim also covers deployment, please report registration and inference costs.
  4. What identity, scenario and query counts remain after exclusion? Are these identical across Table 1 methods and Figure 4 library sizes? If not, please report comparisons on a common population.
  5. How does nearest-adapter routing compare with fixed or random selection from the same bank on the same evaluation population?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.