- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Based on my understanding, here's a summary of the paper:
- The authors demonstrate their pipeline to obtain a new benchmark, GroupRefBench, which is a new benchmark for Multi-S2V generation, evaluating identity, quality, etc of generated videos. Importantly the text prompts don't contain any textual cues about the subjects.
- 100K Training samples(GroupRef100K) for training a Video gen model to evaluate on GroupRefBench.
- The paper proposes CMIA (Cross Modal Identity Anchor) method which adds a learnable embedding to both image and text tokens during training. This is shown later to help in identity binding based from text prompt.
- Results on GroupRefBench show the trained model from GroupRef100K with CMIA improves all identity metrics.
Critical strengths
Here are the strengths of the paper from my reading:
Clarity and Presentation: The paper is very clear in presenting the idea, benchmark curation, as well as demonstrating results. Figures and tables are self-explanatory. The paper reads well. This is important for a reader as it provides the idea in simple terms.
CMIA: The results in Table 4 clearly show that CMIA is highly effective in identity binding and preservation. It is also a simple approach which helps improve identity binding for multiple references. This has the capability of making the method widely usable by the community. Additionally, the motivation of this approach is clear.
Dataset: The need for proposing this benchmark is well-motivated (extending to 10 people, no textual descriptors except the tag).
Critical weaknesses
[W1] Synthetic Faces: The dataset is made of synthetically generated faces. As the authors might be aware, ID preservation for a synthetic face and real face is different in S2I/S2V models. i.e. an S2V model might not be able to preserve a real identity but can easily preserve synthetic identity. The motivation for using synthetic identities is unclear, as the final use-case would be real faces.
[W2] Coverage: As there are no textual identifiers input to the Qwen model for generating the face, the biases from the image generator could easily have been introduced in this dataset. If the Qwen model prefers certain ethnicities, genders or ages, it could very easily affect the data distribution in this data. The authors are suggested to report a data distribution similar to <1>. As Qwen has been probed for generation, is there a possibility of repeated faces appearing in this dataset? There is also a high possibility that a certain ethnic or age group is absent.
[W3] Number of People Analysis: The paper motivates increasing the number of faces (K) to 3-10, however there is no analysis of metrics plotted against values of K. For instance, how do all models compare on K=3,4,5,6,7,8,9,10? How does CMIA affect K=3,4,5,6,7,8,9,10 compared to the baseline?
[W4] Limited Comparisons: A strong benchmark would not only report scores for all methods, but also analyze why certian methods fail at certain axes. This comparison is lacking.
[W5] Quality Drop: Observing the numbers, visual results in the paper and also the supplemental videos provided by the authors, I observed a considerable quality drop in the generated videos compared to the RefAlign baselines.
Questions for authors
[Q1] Are the authors planning to make the data public? This commitment wasn't noted in the paper, as the dataset is part of the contributions.
[Q2] Have the authors considered adding a quality term to the overall loss for training this model? This might help alleviate W5, the quality loss relative to baseline.
[Q3] Why was CMIA considered additive and not rotational? Recent papers in multi-S2I(subject to image generation) generation<2,3> show that adding an extra axis of rope and rotating the tokens corresponding to different identities have helped improve the ID alignment. Have the authors considered ablating on the type of embedding for CMIA?
[Q4] Have the authors considered performing a correlation analysis with OpenS2V or other S2V benchmarks? If it is decorrelated, it will make a good case for the new benchmark as it will show that GroupRef can measure a different axis compared to this.