- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces THQA-LW, a dataset for evaluating an LLM–TTS–T2I–Talker pipeline. It collects human ratings for each component and the overall experience and proposes LT3-FAMT, which combines frozen multimodal features with regression heads to predict these scores.
Critical strengths
Evaluating intermediate outputs alongside the final video could help identify weaknesses in digital-human systems.
Critical weaknesses
-
I noticed that 20 references include Yingjie Zhou, and all five existing benchmark datasets in Section 5.1 come from papers involving this researcher. This is not necessarily a problem, but I would like to understand why these datasets were chosen and whether the results hold on benchmarks outside this group of work. Also, training and testing within each dataset does not show that the model can transfer from one dataset to another.
-
I am not convinced by the contribution as it currently stands. The pipeline connects existing components, and the predictor combines frozen features with regression heads. Although the dataset contains 10,000 videos, they come from only 16 questions, 80 answers, and 20 reference images. I find this coverage quite limited for evaluating interaction quality in general. The regression shows which scores are correlated with overall quality, but it does not tell us whether improving a particular component will actually improve the experience.
-
I am also unclear about the annotation workload. Appendix A reports 868,000 ratings and four million binary decisions from 40 participants. Does this mean every participant evaluated the entire collection? Since many videos are minutes long, how long did the study take, and were participants required to watch each video in full? Please clarify the workload per person, how many ratings were excluded, and how much agreement there was between participants.
Questions for authors
See weakness.