acceptodds
Withdrawn from ICLR 2027

Who is a Better Architect: Subjective and Objective Quality Assessment for Talking Human Interaction Pipeline

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive digital humans are commonly built as cascades of specialized components, known as learnware (LW), offering lower response latency and smaller deployed models relative to omni models. Existing quality assessment, however, focuses primarily on individual learnware and lacks a system-level framework for evaluating the overall interaction. We integrate four learnware central to digital human interaction—a large language model (LLM), text-to-speech (TTS) synthesis, text-to-image (T2I) character generation, and a talking-human (TH) generator, or Talker—into the LT3 pipeline. To study interaction quality from the perspective of learnware, we introduce THQA-LW, a dataset of 10,000 single-turn interaction videos, together with a subjective protocol that separately scores each learnware and the overall interaction experience. We then analyze how learnware quality relates to overall interaction quality under controlled generation conditions. Finally, we propose LT3-FAMT, a five-task quality predictor that combines Qwen2.5-Omni Thinker with complementary multimodal feature extractors. LT3-FAMT achieves state-of-the-art performance across multiple benchmarks, supporting automatic assessment and diagnosis of quality bottlenecks in interactive digital humans.

Primary area:
datasets and benchmarks
Keywords:
ML: Evaluation, Benchmarking, Datasets & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xecfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces THQA-LW, a dataset for evaluating an LLM–TTS–T2I–Talker pipeline. It collects human ratings for each component and the overall experience and proposes LT3-FAMT, which combines frozen multimodal features with regression heads to predict these scores.

Critical strengths

Evaluating intermediate outputs alongside the final video could help identify weaknesses in digital-human systems.

Critical weaknesses
  1. I noticed that 20 references include Yingjie Zhou, and all five existing benchmark datasets in Section 5.1 come from papers involving this researcher. This is not necessarily a problem, but I would like to understand why these datasets were chosen and whether the results hold on benchmarks outside this group of work. Also, training and testing within each dataset does not show that the model can transfer from one dataset to another.

  2. I am not convinced by the contribution as it currently stands. The pipeline connects existing components, and the predictor combines frozen features with regression heads. Although the dataset contains 10,000 videos, they come from only 16 questions, 80 answers, and 20 reference images. I find this coverage quite limited for evaluating interaction quality in general. The regression shows which scores are correlated with overall quality, but it does not tell us whether improving a particular component will actually improve the experience.

  3. I am also unclear about the annotation workload. Appendix A reports 868,000 ratings and four million binary decisions from 40 participants. Does this mean every participant evaluated the entire collection? Since many videos are minutes long, how long did the study take, and were participants required to watch each video in full? Please clarify the workload per person, how many ratings were excluded, and how much agreement there was between participants.

Questions for authors

See weakness.

Reviewer_pJzTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the interesting problem of automatic assessment and diagnosis of quality bottlenecks in interactive digital humans. In the paper, it integrates four modules like LLM, TTS, T2I, and TH into a pipeline and a new dataset of 10000 single-turn interaction videos are introduced. According to the experiments, the proposed LT3-FAMT obtains reasonable results on the benchmarks.

Critical strengths
  • The dataset of THQA-LW is well constructed, which includes 10000 single-turn interaction videos as well as subjective ratings. Different from previous quality assessment, the new dataset provides the assessment of the full interaction pipeline and supports the analysis of how learnware quality relates to the quality.

  • The paper reports reasonable experimental results in Table 3 based on six digital human quality assessment datasets. Also, a set of ablation studies have been included, like Table 4, to validate the setting of the proposed algorithm.

Critical weaknesses
  • There are some magic numbers defined in MOS quality in Eq. 1. How to set there parameters and whether the final results related with the setting of these values?

  • I would have concern on the diversity of the proposed THQA-LW dataset. The dataset is constructed based on 16 single-turn user questions and only 20 reference images are used. Thus, the distribution of the dataset is limited.

  • The evaluation of performance in Table 3 is based a set of metrics like SRCC, PLCC, KRCC, and RMSE. As there metrics may not well reflect the human accessment. How about an user-study of the comparison with the baselines?

  • The presentation of the paper can be improved. For examples, all abbreviations in the paper should be defined at their first occurrence, e.g., SRCC, PLCC, KRCC, and RMSE.

Questions for authors

Please mainly address the issues in teh weakness section.

Reviewer_swqTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses system-level quality assessment for interactive digital humans built as cascades of specialized "learnware" (LW) modules. The authors define the LT3 pipeline(LLM, TTS, T2I, Talker) and construct THQA-LW, a dataset of 10,000 single-turn interaction videos generated from 5 models per LW module (20 models total), with subjective ratings from 40 participants across five quality dimensions (LLM, TTS, T2I, Talker, and overall interaction). They analyze how each module's quality correlates with overall interaction quality via standardized linear regression. They then propose LT3-FAMT, a multitask multimodal quality predictor combining frozen Qwen2.5-Omni Thinker, Jina text embeddings, DINOv2, WavLM, and VideoMAE features with task-specific attention pooling and independent regression heads. LT3-FAMT achieves SOTA on six digital-human QA datasets. Validation includes five-fold cross-validation, ablation studies, and comparison with 19 baseline methods.

Critical strengths
  1. System-level assessment framework for digital-human pipelines. This is the first work to systematically evaluate quality at each stage of a cascaded digital-human pipeline rather than only the final video. The LT3 pipeline abstraction and the per-LW subjective protocol are consequential because they enable bottleneck diagnosis—architects can now identify which module degrades the interaction, not just that it is degraded. This directly addresses a real gap in the literature, as confirmed by Table 1 showing no prior database includes LW-level analysis.
  2. Large-scale dataset with multidimensional annotations. THQA-LW's 10,000 videos with traceable provenance (linked to specific models, voices, prompts, and configurations) and five-dimensional quality labels is a substantial community resource. The matched design (5 Talkers processing the same audio-image pairs) enables controlled comparisons that prior datasets cannot support.
Critical weaknesses
  1. The LW-to-overall attribution analysis is methodologically too simplistic to support strong causal claims. The standardized linear regression in Eq. (1) is the paper's key analytical contribution, yet it assumes independence and additivity among LW modules. The paper's own distortion analysis (Section 3.5) shows that most videos exhibit multiple co-occurring distortion types, directly contradicting the independence assumption. Without interaction terms or a non-linear model, the reported coefficient dominance of Talker (β=0.838) may be overstated— Talker distortions are simply the most visible in the final product, while upstream effects (e.g., an irrelevant LLM answer) could have non-linear or threshold effects on perceived quality. This is consequential because the paper uses this result to justify focusing diagnostic attention on the Talker, which may be an artifact of the model choice.
  2. Narrow content diversity limits the generalizability of the core findings. The entire dataset is built from only 16 questions. The paper claims the dataset covers "a broad range of topics" but provides no quantitative or qualitative analysis of topic coverage. With 4 questions per interaction purpose, the space of user intents is severely under-sampled. Since LLM quality is one of the five dimensions being assessed, and LLM answers are entirely determined by the 16 input questions, the LLM quality ratings may reflect question-specific artifacts rather than general LLM capability differences. This weakens the dataset's utility for benchmarking and the validity of the regression coefficients.
Questions for authors

1.Attribution model robustness: Have you tried non-linear models (e.g., random forest, gradient boosting, or GAM) or regression with cross-terms for the LW-to-overall quality attribution? If so, how do the relative importances change? If not, why did you settle on a linear model given the known compound distortion patterns? 2.Question diversity: How sensitive are the regression coefficients (especially LLM's near-zero β=0.012) to the choice of the 16 questions? Have you tested whether adding more questions changes the relative importance of LW modules? A leave-one-question-out analysis on the regression coefficients would address this. 3. Effective ratings per video: After observer screening, what is the average and minimum number of valid ratings per video across the five dimensions? Does the reliability of per-video MOS differ across dimensions due to differential screening? 4. Per-LW branch evaluation on external data: Is there any way to validate the per-LW quality prediction branches on data other than THQA-LW? For instance, could you use THQA-10K or THQA-NTIRE videos and retrospectively estimate upstream module quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.