- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a new benchmark to evaluate omnimodal personalization across images, audio and text. The benchmarking dataset includes 750 items for 4 tasks and 18 subtasks. Around half of the items omit the target person. The paper uses Calibrated Accuracy to jointly measure correct answering and abstention. The experiments reveal weaker audio than visual grounding, inconsistent benefits from scaling SFT data, and a trade-off in which RLVR improves answerable recall but reduces appropriate abstention.
Critical strengths
- The paper is well-written and easy to follow.
- The new benchmark has a good motivation.
- There are some analysis based on the benchmarking dataset, which provides some guidance.
Critical weaknesses
- The benchmark contains only 750 items across 18 subtasks. This leaves relatively few examples per subtask, making the analysis based on the benchmark not quite convincing.
- In Table S.2, removing both images and audio improves T2Any accuracy from 52% to 62% for Qwen and from 57% to 72% for Gemma. This raises concerns that these tasks can be solved through textual shortcuts. The authors should clarify which tasks require non-textual evidence and validate their cross-modal dependence.
- The results do not clearly distinguish between identifying the wrong person and answering incorrectly about the right person. The reward ablations provide some insight, but separate identity-matching scores and QA results with the correct persona provided would make the analysis more informative.
Questions for authors
- Since the benchmark only include 750 items, sample counts and confidence intervals would help assess whether the reported differences are reliable.
- Please separate identity-matching scores and QA results with the correct persona provided to separate different failure types.