acceptodds
Under review as a conference paper at ICLR 2027

Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization

Abstract

While multimodal large language models have advanced across text, image, and audio, personalization research has remained primarily vision-language, with unified omnimodal benchmarking that jointly covers text, image, and audio still limited and lacking methodological rigor for absent-persona scenarios and systematic grounding studies. We introduce Omni-Persona, the first comprehensive benchmark for omnimodal personalization. We formalize the task as cross-modal routing over the Persona Modality Graph, encompassing 4 task groups and 18 fine-grained tasks across items. To rigorously diagnose grounding behavior, we propose Calibrated Accuracy (), which jointly evaluates correct grounding and appropriate abstention by incorporating absent-persona queries into a unified evaluation framework. Our experiments reveal three findings: (i) recent open-weight models exhibit a consistent audio-versus-visual grounding gap that RLVR partially narrows through dense rule-based supervision; (ii) recall and parameter scale are incomplete diagnostics, as strong recall can coexist with absent-persona hallucination and larger models do not always achieve higher , establishing calibration as a distinct evaluation axis; and (iii) SFT is constrained by the scalability of ground-truth annotation. Although RLVR improves recall, it does not improve calibrated abstention under our reward design, leaving at or below the base model. Omni-Persona therefore provides a diagnostic framework for identifying pitfalls in omnimodal personalization and guiding future post-training and reward design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omnimodal personalization, persona memory graph, absent persona

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1AFUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new benchmark to evaluate omnimodal personalization across images, audio and text. The benchmarking dataset includes 750 items for 4 tasks and 18 subtasks. Around half of the items omit the target person. The paper uses Calibrated Accuracy to jointly measure correct answering and abstention. The experiments reveal weaker audio than visual grounding, inconsistent benefits from scaling SFT data, and a trade-off in which RLVR improves answerable recall but reduces appropriate abstention.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The new benchmark has a good motivation.
  3. There are some analysis based on the benchmarking dataset, which provides some guidance.
Critical weaknesses
  1. The benchmark contains only 750 items across 18 subtasks. This leaves relatively few examples per subtask, making the analysis based on the benchmark not quite convincing.
  2. In Table S.2, removing both images and audio improves T2Any accuracy from 52% to 62% for Qwen and from 57% to 72% for Gemma. This raises concerns that these tasks can be solved through textual shortcuts. The authors should clarify which tasks require non-textual evidence and validate their cross-modal dependence.
  3. The results do not clearly distinguish between identifying the wrong person and answering incorrectly about the right person. The reward ablations provide some insight, but separate identity-matching scores and QA results with the correct persona provided would make the analysis more informative.
Questions for authors
  1. Since the benchmark only include 750 items, sample counts and confidence intervals would help assess whether the reported differences are reliable.
  2. Please separate identity-matching scores and QA results with the correct persona provided to separate different failure types.
Reviewer_rU3sOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new benchmark for personalization in multimodal model: Omni-Persona The main argument is that prior work often focus on text <-> image only, while this newly introduced Omni-Persona include more modality: (i) text, (ii) image, and (iii) audio.

The benchmark is synthesized and has total of 18 tasks, with about 450 distinct person. Evaluation of multiple frontier model (e.g., Gemini) and some open-sourced models (e.g., Qwen), together with SFT and RL these open-sourced models results.

Critical strengths

I agree with the motivation that audio modality is lacking in the current personalization of MLLMs domain. I appreciate the efforts to create the benchmark with extended modality: text <-> audio <-> image.

Critical weaknesses

While I appreciate the efforts and really like the direction, I feel the current benchmark lacks of deeper personalized understanding -- it's more like coarse appearance cues. With the current personalization field is moving from "recognization" to "understanding" -- I am unsure if readers can gain many interesting points from the current submission.

  • I think the downstream tasks are too naive: matching a given audio/text/image to an identity. It's more like testing "recognition" abilities rather than “understanding” abilities. For example, in Figure S.9, matching Jacob’s image to the text saying "He seemed calm and contemplative, a little distant" doesn't clearly show the benefit of having all three modalities. love the idea, yet I believe it would be much stronger if it focused on, let's say, two modalities first (e.g., video <-> audio), with tasks that require deeper personalized understanding. For example: video understanding with personalized subjects, where you need to identify who is in the scene, who is talking, and use what you know about them to understand their interactions.
  • The candidate pool is too limited to clearly tell if the model can reliably distinguish identities. For example, how many very similar-looking people are there within each example (e.g., two Asian men with short hair and similar appearances)? The paper mentions hard distractors, but it's unclear how visually similar they actually are. Thus, I’m concerned that the current matching may rely on coarse demographic or appearance cues rather than actual identity matching. I’d suggest testing with larger candidate pools and more controlled, similar-looking distractors.
  • Another minor issue about the benchmark is that, these identities in the benchmark might be famous people, which model can already recognize them (e.g., famous actress) -- Which could make the identity matching easier. I'd suggest clarify the identities source.
Questions for authors
  • Please clarify how many identities are there in the dataset? And the distribution of this (e.g., famous vs. non-famous person, in which famous person = the current VLMs can identify their name).
  • The finding in the experiments is that there is a "recall-abstention trade-off": RL improves answerable accuracy, yet also encourages the model to answer without enough evidence (or guess). I'm not sure how specific this insight is to omnimodal personalization, rather than a consequence of the current RL reward design. I appreciate the discussion and ablations, but I believe the paper would be stronger with a clearer explanation of what causes this trade-off, or an improvement to the RL training that reduces unsupported answers while preserving the accuracy gains.
Reviewer_QQUZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies omnimodal personalization across text, image, and audio and proposes the Omni-Persona benchmark. It introduces the Persona Modality Graph (PMG) to model cross-modal matching and grounding, covering four task groups and 18 fine-grained tasks with about 750 samples. The paper proposes Calibrated Accuracy (Cal) to evaluate correct answering and abstention when the target persona is absent. The experiments compare several open-source and closed-source models and examine the effects of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). The results show that visual grounding is generally stronger than audio grounding in open-source models, while larger models do not always perform better. SFT provides limited improvements, while RLVR improves answerable recall but yields lower Cal scores across the tested backbones, with the decreases not statistically significant for the two Qwen models examined in Appendix F.2.

Critical strengths
  1. The paper clearly defines the problem of omnimodal personalization across text, image, and audio. It identifies the limits of audio grounding and the lack of evaluation for absent-persona cases, making the research motivation clear.
  2. The benchmark has a clear structure. The Persona Modality Graph (PMG) organizes four task groups and 18 fine-grained tasks. Calibrated Accuracy (Cal) evaluates both correct answering and proper abstention, which helps identify problems missed by recall-only metrics.
  3. The experiments cover several open-source and closed-source models and compare SFT with RLVR. The paper also includes reward ablations and additional analyses, which help explain the differences between visual and audio grounding and the trade-off between recall and abstention.
Critical weaknesses
  1. The benchmark focuses on grounding with pre-retrieved persona contexts, but it does not evaluate the retrieval process itself. Also, much of the persona data is synthetic. These settings limit how well the results reflect real-world personalization systems.
  2. The benchmark uses LLMs to impute missing persona traits. This may introduce unverified personal information and affect the reliability of ground-truth labels. The distinction between source-supported facts and generated traits needs clearer explanation.
  3. The related work could be expanded with recent studies on cross-modal representation learning and multimodal fusion. These studies may help explain how information from different modalities can be aligned and combined. For example, the authors may consider the following papers: Yu et al., "CAMeL: Cross-Modality Adaptive Meta-Learning for Text-Based Person Retrieval," IEEE Transactions on Information Forensics and Security, 2025;
Questions for authors
  1. The paper formalizes cross-modal personalization as routing over the Persona Modality Graph (PMG). Beyond organizing the 18 tasks, what does the graph formulation add to the evaluation, especially for absent-persona cases?
  2. In Table S.6, removing the localization reward increases Cal to 45.7% but reduces overall recall to 31.5%. How should this trade-off be interpreted when evaluating model performance?
  3. The evaluation uses keyword matching to identify abstentions. How do the authors handle valid abstentions that use different wording? Could they briefly discuss how this affects the reliability of Cal?
  4. The benchmark uses LLM-generated persona descriptions and imputes missing attributes. How do the authors distinguish source-supported attributes from imputed traits when assigning ground-truth labels?
  5. The SFT and RLVR experiments use different training tasks and supervision methods. How do the authors separate the effects of the training method from the differences in task coverage when comparing their results?
  6. The RLVR reward design includes rewards for correct abstention, but the trained models still show lower true-abstention rates. Could the authors explain this behavior using the reward and ablation results already reported in the appendix?
  7. The paper reports lower Cal scores after RLVR on all four backbones. However, the Cal changes are not statistically significant for the two Qwen models tested in Appendix F.2. Could the authors clarify the strength of this conclusion and reflect the statistical results in the main discussion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.