- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
Problem and objective. The paper addresses evaluation of generated audio-video content under diverse conditioning inputs. Its central premise is that identifying defects and determining human preference are distinct problems: shared defects may cancel in a comparison, while different defects have different importance. JACE aims to connect these problems without requiring defect annotations and preference labels on the same videos.
-
Method and contributions. JACE first learns 46-category defect predictions from incomplete annotations using frozen Qwen3-Omni representations, then applies category-specific residual corrections using auxiliary representations. The diagnostic module is subsequently frozen, and a separate preference collection trains an independent quality head and nonnegative defect weights. Their combined score supports individual-video diagnosis, decomposable pairwise comparisons with ties, and Bradley–Terry generator ranking.
-
Validation. Experiments cover text-to-video, image-to-video, reference-to-video, and video editing. JACE reports 56.52% defect Recall@3, 69.52% non-tie accuracy, 58.31% three-way GSB accuracy, and 0.8182 Kendall ranking agreement. Comparisons include specialized evaluators, general multimodal models, and matched representation readouts. Ablations examine incomplete-label handling, residual correction, and preference weighting, although annotation reliability and unseen-generator generalization remain insufficiently documented.
Critical strengths
-
The separation of diagnosis and preference is well motivated and practically useful. Defect recognition alone cannot determine which candidate humans prefer, especially when both candidates share a defect or exhibit different quality trade-offs. JACE explicitly learns diagnosis and preference weighting from separate video collections, allowing the two annotation workflows to scale independently. This formulation is a substantive strength because it addresses a realistic supervision constraint while connecting diagnostic evidence to overall evaluation.
-
Controlled ablations provide evidence for the principal design choices. Residual correction improves task-mean Recall@3 from 42.8% to 56.5%, outperforming prediction-score fusion and feature concatenation. Combining quality and attribution improves GSB accuracy from 54.0% for quality-only scoring to 58.3%, and learned defect weights outperform equal weighting by approximately two percentage points. These comparisons are consequential because they support the contributions of correction and preference calibration beyond simply selecting a stronger encoder.
-
The scoring formulation provides inspectable comparisons across multiple tasks. Category contributions are explicit terms in the score difference rather than separately generated explanations, and the comparison formulation preserves win/loss symmetry when candidates are swapped. The same mechanism is evaluated across four generation tasks and connected to generator ranking. This combination offers useful methodological coherence: users can inspect the evidence contributing to a decision while retaining a common scoring framework across evaluation settings.
Critical weaknesses
-
The evidence does not adequately establish generalization to unseen generators. The paper specifies prompt- and media-association-based partitions, but does not clearly identify the generator families represented in each partition or report a generator-held-out evaluation. Content separation does not rule out learning familiar generator-specific artifacts. Because cross-generator comparison is a central application, this omission limits confidence that the evaluator remains reliable when applied to newly released or substantially different generation models.
-
The human supervision underlying the evaluation is insufficiently characterized. The paper reports annotation counts but provides limited information about annotator numbers, agreement, adjudication, and the operational definition of a tie. These details matter particularly because defect labels are deliberately non-exhaustive and preference labels determine category weights. Without a clearer account of label reliability, it is difficult to distinguish agreement with stable human judgments from agreement with the particular annotation process used to construct the evaluation set.
-
Diagnostic reliability and the robustness of smaller gains remain uncertain. Recall@3 measures retrieval of recorded defects and excludes overall-good examples from its denominator; it therefore does not establish diagnostic precision or false-positive behavior on good videos. Meanwhile, the additional preference benefit of learned weights over equal weights is approximately two percentage points, without reported uncertainty estimates. Since reliable diagnosis and effective weighting are central contributions, precision audits and statistical uncertainty could materially change how much confidence should be placed in these improvements.
Questions for authors
-
Clarification on generator coverage and generalization: Which generator families and versions contributed videos to each training, calibration, and evaluation collection? Were any generators entirely absent from both diagnostic and preference training, and, if so, what results were obtained on them? Existing held-out results would materially strengthen the claimed applicability to cross-generator evaluation.
-
Clarification on human annotation reliability: How many annotators assessed each video or pair, what instructions defined defects and ties, and how were disagreements resolved? Please report any available agreement statistics and explain how overall-good labels were verified despite the non-exhaustive defect annotation protocol. These details are necessary to assess the reliability of both supervision sources.
-
Clarification on diagnostic precision and result uncertainty: Are there existing manual audits or exhaustively labeled subsets that measure false-positive defects, particularly on overall-good videos? Please also provide any available confidence intervals or prompt-group bootstrap analyses for the learned-weight versus equal-weight comparison and generator-ranking agreement. Such evidence would determine whether the diagnostic outputs and smaller calibration gains are sufficiently reliable.
-
Presentation suggestion: I would appreciate more qualitative visualizations in the main paper beyond the single comparison in Figure 4. Examples spanning different generation tasks, including an audio-related defect, a tie, and a failure case, would help readers assess whether the predicted defects are grounded in the content and how their contributions affect the final judgment. This is a suggestion to improve presentation rather than a prerequisite for acceptance.
2 comments
Response to Reviewer hoQU (Part 2/2)
W3 / Q3. Defect false positives and stability of the results
False positives on overall-good videos. We randomly sampled 25 videos from each of the four tasks in the attribution confirmation set, for a total of 100 videos. With the original labels and model outputs hidden, reviewers first judged whether each video contained a salient defect. For videos they judged to contain a defect, they then checked each of JACE’s three highest-ranked defect predictions. Of the 18 videos that reviewers labeled GOOD, JACE flagged 2 as having a salient defect (2/18) and classified the remaining 16 as GOOD. JACE classified 22 videos as GOOD in total. We calculated defect Precision@3 on the 82 videos with human-confirmed defects. Reviewers confirmed 156 of the 246 predicted defects.
| Method | Defect false-positive rate on GOOD videos (%) | Defect Precision@3 (%) | GOOD precision (%) | GOOD recall (%) | GOOD F1 (%) |
|---|---|---|---|---|---|
| JACE | 11.1 | 63.4 | 72.7 | 88.9 | 80.0 |
Gain from learning defect weights. With the quality scoring head and defect predictions held fixed, overall GSB accuracy increased from 56.3% with equal weights to 58.3% with learned weights (Table 3). The 95% confidence interval for the difference was [+1.07, +2.83] percentage points. The gains for T2V, I2V, and V2V were +3.13, +1.56, and +2.34 percentage points, respectively, with all three confidence intervals lying above zero. R2V showed a gain of 0.78 percentage points, with a confidence interval that included zero.
Stability of generator rankings. Table 1 reports agreement between JACE’s ranking of the generators and the human ranking (Kendall’s τᵦ = 0.8182). To assess sensitivity to prompt sampling, we drew 5,000 bootstrap samples by resampling prompts with replacement. In each replicate, we retained all pairwise comparisons for each sampled prompt, refitted the human and JACE Bradley–Terry rankings separately, and recomputed τᵦ. The resulting 95% percentile confidence interval was [0.6364, 0.9636], indicating that the two rankings remained positively correlated as the sampled prompts varied.
Q4. Additional qualitative examples
Thank you for the suggestion. We will include a wider range of qualitative examples in the revised manuscript to illustrate JACE’s judgments across generation tasks.
Response to Reviewer hoQU (Part 1/2)
W1 / Q1. Generator coverage and generalization to unseen generators
JACE’s training data span several generator families. SD2.0 and Wan2.7 together account for at least 80% of the training data, which also include versions of Kling and HappyHorse. The evaluation set covers 11 video generators, including three newer models unseen during training: SD2.5, Wan3.0, and MiniMax-H3. The original manuscript reported results pooled across generators. Following the reviewer’s suggestion, we examined performance on the newer models separately. We compared SD2.5 with SD2.0 and Wan3.0 with Wan2.7 to assess changes within each family. MiniMax-H3 provides a test on a new family.
JACE’s accuracy in predicting human win, tie, and loss judgments was similar for SD2.0 and SD2.5, with a difference of 0.79 percentage points. Accuracy dropped by 4.69 percentage points from Wan2.7 to Wan3.0. On MiniMax-H3, JACE achieved 58.59% GSB accuracy, close to the four-task mean of 58.31% in Table 6. This result provides some evidence that JACE generalizes to new generators.
W2 / Q2. Reliability of human annotations
Single-video defect annotation. Three annotators assessed each video for salient defects against its generation prompt and any reference inputs. A fourth person reviewed the annotations. Approximately 80–100 people participated in annotation and quality control. The GOOD label indicates that no salient defect was observed. When annotators disagreed about a defect or the GOOD label, the verifier determined the final label. Because subtle defects can go unrecorded, we use GOOD as a weaker negative reference during training. We also sampled videos from all four tasks and had reviewers independently assess whether a salient defect was present, with both the original labels and the model outputs hidden.
Pairwise preference evaluation. In the formal evaluation, 3–5 raters judged each anonymized video pair, with one vote per rater. They compared the overall quality of two candidates generated under the same conditions and selected an A win, a B win, or a tie. Raters could choose a tie when the candidates were comparable in quality, including when both were good or both were poor. We aligned votes to a common A/B order before tallying the original three choices. When an A win and a B win were tied for the most votes, we recorded the comparison as a tie. For video pairs with multiple raters who each cast one vote, we calculated the proportion of rater pairs that gave the same three-way judgment, then averaged this proportion across video pairs.