acceptodds
Under review as a conference paper at ICLR 2027

JACE: Judge with Attribution, Calibration, and Evaluation for Multimodal Audio-Video Generation

Abstract

Video generation increasingly combines multimodal conditions to produce visual and auditory content, placing greater demands on evaluation. Existing evaluators struggle to jointly support diverse generation tasks and provide interpretable judgments aligned with human preferences. Accurate defect prediction requires understanding physical plausibility, temporal coherence, and cross-modal consistency, which is challenging. Even accurate diagnosis, however, does not determine preference: shared defects may offer little basis for comparison, while different defects may carry different importance. We introduce JACE (Judge with Attribution, Calibration, and Evaluation), a generalist evaluator that connects defect diagnosis to human preference through explicit scoring contributions. JACE first learns defect predictions from non-exhaustive annotations, using complementary representations for category-specific refinement. It then freezes the diagnostic module and learns defect-category weights from separately collected pairwise preferences, without requiring both annotation types on the same videos. Combining weighted defect contributions with a base quality score yields interpretable win, tie, and loss judgments. JACE supports text, image, audio, and video conditioning across text-to-video, image-to-video, reference-to-video, and video editing tasks, jointly assessing generated visuals and sound. Its shared scoring mechanism connects individual-video diagnosis and scoring, candidate comparison, and generator ranking, supporting evaluation across generators. Across four generation tasks, JACE significantly improves defect Recall@3 (+24.6%) and non-tie accuracy by (+8.09%) over the strongest baseline (Gemini 3.7 Flash).

Primary area:
generative models
Keywords:
video generation, sparse attribution, preference judgment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hoQUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Problem and objective. The paper addresses evaluation of generated audio-video content under diverse conditioning inputs. Its central premise is that identifying defects and determining human preference are distinct problems: shared defects may cancel in a comparison, while different defects have different importance. JACE aims to connect these problems without requiring defect annotations and preference labels on the same videos.

  2. Method and contributions. JACE first learns 46-category defect predictions from incomplete annotations using frozen Qwen3-Omni representations, then applies category-specific residual corrections using auxiliary representations. The diagnostic module is subsequently frozen, and a separate preference collection trains an independent quality head and nonnegative defect weights. Their combined score supports individual-video diagnosis, decomposable pairwise comparisons with ties, and Bradley–Terry generator ranking.

  3. Validation. Experiments cover text-to-video, image-to-video, reference-to-video, and video editing. JACE reports 56.52% defect Recall@3, 69.52% non-tie accuracy, 58.31% three-way GSB accuracy, and 0.8182 Kendall ranking agreement. Comparisons include specialized evaluators, general multimodal models, and matched representation readouts. Ablations examine incomplete-label handling, residual correction, and preference weighting, although annotation reliability and unseen-generator generalization remain insufficiently documented.

Critical strengths
  1. The separation of diagnosis and preference is well motivated and practically useful. Defect recognition alone cannot determine which candidate humans prefer, especially when both candidates share a defect or exhibit different quality trade-offs. JACE explicitly learns diagnosis and preference weighting from separate video collections, allowing the two annotation workflows to scale independently. This formulation is a substantive strength because it addresses a realistic supervision constraint while connecting diagnostic evidence to overall evaluation.

  2. Controlled ablations provide evidence for the principal design choices. Residual correction improves task-mean Recall@3 from 42.8% to 56.5%, outperforming prediction-score fusion and feature concatenation. Combining quality and attribution improves GSB accuracy from 54.0% for quality-only scoring to 58.3%, and learned defect weights outperform equal weighting by approximately two percentage points. These comparisons are consequential because they support the contributions of correction and preference calibration beyond simply selecting a stronger encoder.

  3. The scoring formulation provides inspectable comparisons across multiple tasks. Category contributions are explicit terms in the score difference rather than separately generated explanations, and the comparison formulation preserves win/loss symmetry when candidates are swapped. The same mechanism is evaluated across four generation tasks and connected to generator ranking. This combination offers useful methodological coherence: users can inspect the evidence contributing to a decision while retaining a common scoring framework across evaluation settings.

Critical weaknesses
  1. The evidence does not adequately establish generalization to unseen generators. The paper specifies prompt- and media-association-based partitions, but does not clearly identify the generator families represented in each partition or report a generator-held-out evaluation. Content separation does not rule out learning familiar generator-specific artifacts. Because cross-generator comparison is a central application, this omission limits confidence that the evaluator remains reliable when applied to newly released or substantially different generation models.

  2. The human supervision underlying the evaluation is insufficiently characterized. The paper reports annotation counts but provides limited information about annotator numbers, agreement, adjudication, and the operational definition of a tie. These details matter particularly because defect labels are deliberately non-exhaustive and preference labels determine category weights. Without a clearer account of label reliability, it is difficult to distinguish agreement with stable human judgments from agreement with the particular annotation process used to construct the evaluation set.

  3. Diagnostic reliability and the robustness of smaller gains remain uncertain. Recall@3 measures retrieval of recorded defects and excludes overall-good examples from its denominator; it therefore does not establish diagnostic precision or false-positive behavior on good videos. Meanwhile, the additional preference benefit of learned weights over equal weights is approximately two percentage points, without reported uncertainty estimates. Since reliable diagnosis and effective weighting are central contributions, precision audits and statistical uncertainty could materially change how much confidence should be placed in these improvements.

Questions for authors
  1. Clarification on generator coverage and generalization: Which generator families and versions contributed videos to each training, calibration, and evaluation collection? Were any generators entirely absent from both diagnostic and preference training, and, if so, what results were obtained on them? Existing held-out results would materially strengthen the claimed applicability to cross-generator evaluation.

  2. Clarification on human annotation reliability: How many annotators assessed each video or pair, what instructions defined defects and ties, and how were disagreements resolved? Please report any available agreement statistics and explain how overall-good labels were verified despite the non-exhaustive defect annotation protocol. These details are necessary to assess the reliability of both supervision sources.

  3. Clarification on diagnostic precision and result uncertainty: Are there existing manual audits or exhaustively labeled subsets that measure false-positive defects, particularly on overall-good videos? Please also provide any available confidence intervals or prompt-group bootstrap analyses for the learned-weight versus equal-weight comparison and generator-ranking agreement. Such evidence would determine whether the diagnostic outputs and smaller calibration gains are sufficiently reliable.

  4. Presentation suggestion: I would appreciate more qualitative visualizations in the main paper beyond the single comparison in Figure 4. Examples spanning different generation tasks, including an audio-related defect, a tie, and a failure case, would help readers assess whether the predicted defects are grounded in the content and how their contributions affect the final judgment. This is a suggestion to improve presentation rather than a prerequisite for acceptance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hoQU (Part 1/2)

W1 / Q1. Generator coverage and generalization to unseen generators

JACE’s training data span several generator families. SD2.0 and Wan2.7 together account for at least 80% of the training data, which also include versions of Kling and HappyHorse. The evaluation set covers 11 video generators, including three newer models unseen during training: SD2.5, Wan3.0, and MiniMax-H3. The original manuscript reported results pooled across generators. Following the reviewer’s suggestion, we examined performance on the newer models separately. We compared SD2.5 with SD2.0 and Wan3.0 with Wan2.7 to assess changes within each family. MiniMax-H3 provides a test on a new family.

Generator versionGSB accuracyDifference from the earlier version in the same family (percentage points)
SD2.059.38%Reference
SD2.558.59%−0.79
Wan2.761.72%Reference
Wan3.057.03%−4.69
MiniMax-H358.59%—

JACE’s accuracy in predicting human win, tie, and loss judgments was similar for SD2.0 and SD2.5, with a difference of 0.79 percentage points. Accuracy dropped by 4.69 percentage points from Wan2.7 to Wan3.0. On MiniMax-H3, JACE achieved 58.59% GSB accuracy, close to the four-task mean of 58.31% in Table 6. This result provides some evidence that JACE generalizes to new generators.


W2 / Q2. Reliability of human annotations

Single-video defect annotation. Three annotators assessed each video for salient defects against its generation prompt and any reference inputs. A fourth person reviewed the annotations. Approximately 80–100 people participated in annotation and quality control. The GOOD label indicates that no salient defect was observed. When annotators disagreed about a defect or the GOOD label, the verifier determined the final label. Because subtle defects can go unrecorded, we use GOOD as a weaker negative reference during training. We also sampled videos from all four tasks and had reviewers independently assess whether a salient defect was present, with both the original labels and the model outputs hidden.

Pairwise preference evaluation. In the formal evaluation, 3–5 raters judged each anonymized video pair, with one vote per rater. They compared the overall quality of two candidates generated under the same conditions and selected an A win, a B win, or a tie. Raters could choose a tie when the candidates were comparable in quality, including when both were good or both were poor. We aligned votes to a common A/B order before tallying the original three choices. When an A win and a B win were tied for the most votes, we recorded the comparison as a tie. For video pairs with multiple raters who each cast one vote, we calculated the proportion of rater pairs that gave the same three-way judgment, then averaged this proportion across video pairs.

AuthorsOpenReview ↗

Response to Reviewer hoQU (Part 2/2)

W3 / Q3. Defect false positives and stability of the results

False positives on overall-good videos. We randomly sampled 25 videos from each of the four tasks in the attribution confirmation set, for a total of 100 videos. With the original labels and model outputs hidden, reviewers first judged whether each video contained a salient defect. For videos they judged to contain a defect, they then checked each of JACE’s three highest-ranked defect predictions. Of the 18 videos that reviewers labeled GOOD, JACE flagged 2 as having a salient defect (2/18) and classified the remaining 16 as GOOD. JACE classified 22 videos as GOOD in total. We calculated defect Precision@3 on the 82 videos with human-confirmed defects. Reviewers confirmed 156 of the 246 predicted defects.

MethodDefect false-positive rate on GOOD videos (%)Defect Precision@3 (%)GOOD precision (%)GOOD recall (%)GOOD F1 (%)
JACE11.163.472.788.980.0

Gain from learning defect weights. With the quality scoring head and defect predictions held fixed, overall GSB accuracy increased from 56.3% with equal weights to 58.3% with learned weights (Table 3). The 95% confidence interval for the difference was [+1.07, +2.83] percentage points. The gains for T2V, I2V, and V2V were +3.13, +1.56, and +2.34 percentage points, respectively, with all three confidence intervals lying above zero. R2V showed a gain of 0.78 percentage points, with a confidence interval that included zero.

Stability of generator rankings. Table 1 reports agreement between JACE’s ranking of the generators and the human ranking (Kendall’s τᵦ = 0.8182). To assess sensitivity to prompt sampling, we drew 5,000 bootstrap samples by resampling prompts with replacement. In each replicate, we retained all pairwise comparisons for each sampled prompt, refitted the human and JACE Bradley–Terry rankings separately, and recomputed τᵦ. The resulting 95% percentile confidence interval was [0.6364, 0.9636], indicating that the two rankings remained positively correlated as the sampled prompts varied.


Q4. Additional qualitative examples

Thank you for the suggestion. We will include a wider range of qualitative examples in the revised manuscript to illustrate JACE’s judgments across generation tasks.

Reviewer_qnAZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes JACE with attribution, calibration, and evaluation, designing a framework combining defect prediction and preference determination. It first learns defect predictions from non-exhaustive annotations. It then freezes the diagnostic module and learns defect-category weights from separately collected pairwise preferences. At last, it obtains interpretable win, tie, and loss judgments combining weighted defect contributions with a base quality score. It supports text, image, audio, and video conditioning across text-to-video, image-to-video, reference-to-video, and video editing tasks, jointly assessing generated visuals and sound. Across four generation tasks, it significantly improves defect Recall@3 (+24.6%) and non-tie accuracy by (+8.09%) over the strongest baseline (Gemini 3.7 Flash).

Critical strengths

This paper separates defect attribution from preference calibration and allows the use of disjoint annotation datasets. This two-stage learning framework is well motivated and the problem is practical and interesting.

Critical weaknesses
  1. The paper suffers from several presentation issues that substantially affect its clarity and readability. Many sentences are unnecessarily long and densely packed with technical concepts, making the main arguments difficult to follow.
  2. Although the method avoids directly treating all unannotated defects as negative labels, its unknown-category regularization still suppresses predictions for unobserved categories, potentially introducing systematic bias when genuine defects are omitted from annotations, particularly for rare or subtle defect categories.
Questions for authors
  1. In fact, JACE proposes a new reward model. However, no experimental evidence is proposed to evaluate its optimization effect.
  2. As shown in Eq.4 (Sec 4.4), the preference calibration module learns a single set of category-level defect weights w_k shared across samples and generation tasks. This design may limit the model's ability to capture context-dependent human preferences and generalize to different tasks.
  3. As shown in Appendix C.3,the evaluation relies primarily on Recall@3 and excludes videos without observed defects from its calculation, making it difficult to assess false-positive predictions, diagnostic precision, and the ability to correctly recognize videos without salient defects; additional metrics such as precision, F1, or mAP would provide a more complete evaluation.
  4. Although the additive scoring function provides a mathematically decomposable explanation of each judgment, this property does not establish that the estimated defect contributions reflect the factors actually used by human evaluators. Could the authors propose additional human evaluation?
  5. As JACE can propose p_A, p_B, p_T, why the paper does not propose ECE results besides GSB, Non-tie Acc?
  6. Although the paper evaluates JACE on multiple video generation tasks and reports generator-level ranking results, the experimental setup does not clearly specify which video generators are included.
  7. The paper provides meaningful controlled comparisons of pretrained representations under matched supervision in Table 6. However, the generalizability of the proposed methodology across different backbone representations remains unverified. The complete JACE model is evaluated only with the specific combination of Qwen3-Omni and PE-AV, making it difficult to determine whether the improvements arise from the proposed JACE or depend heavily on this particular feature combination.
5 comments
AuthorsOpenReview ↗

Response to Reviewer qnAZ (Part 1/5)

Dear Reviewer qnAZ,

Thank you for acknowledging our contributions and your valuable comments. We address your concern as follows:


W1 — Clarity of the writing

Thank you for your suggestion on how to make the paper clearer. We can explain the motivation for JACE's two-stage design more directly in the introduction. Once we know which defects are present in two videos, how do we decide which video viewers will prefer? In Figure 1, both videos have a watermark. If the watermark incurs the same penalty in both, its contribution cancels out in the comparison. Video A is slightly blurry, while in video B a ball passes through the floor. To predict which video viewers will prefer, we also need to know how much each of these defects matters. JACE therefore first learns to identify defects from annotations on individual videos. We then freeze the diagnostic module and learn a weight for each defect category from pairwise preferences on a separate set of videos. In the revision, we will use the example in Figure 1 to introduce the two stages, split the long sentences that introduce several concepts at once, and specify what annotations each stage uses and what it learns.


W2 — Missing defect annotations and regularization of unknown categories under weak supervision

  1. Why we use weak supervision. JACE's defect taxonomy covers 46 categories. For each video, three annotators label defects and a fourth person reviews the annotations for accuracy. Exhaustive annotation would require checking every applicable category for every video. To make annotation feasible at this scale, we ask annotators to record the salient defects they observe and preserve uncertainty about the remaining categories during training. A GOOD label means that the annotators observed no salient defect, so these videos provide only weak negative evidence.

  2. What weak supervision contributes. Table 2 compares supervision strategies using the same training videos, sparse annotations, and base diagnostic model. Treating every unlabeled category as a negative gives a task-averaged Recall@3 of 38.9% and a category-averaged Recall@3 of 21.2%. JACE treats annotated defects as positives and gives low weight to unlabeled applicable categories in GOOD videos when using them as negatives. In videos labeled non-GOOD with at least one annotated defect, the remaining unlabeled applicable categories are treated as unknown, and their predicted scores are only weakly constrained. This strategy raises the two metrics to 42.8% and 24.8%, gains of 3.9 and 3.6 percentage points, respectively. With the same annotation effort, the model learns to identify more of the recorded defects.

  3. How regularization handles missing annotations. As you point out, if rare or subtle defects are missed more often, their predicted scores may be pushed down more frequently. To mitigate this effect, we treat the remaining unlabeled applicable categories in non-GOOD videos with annotated defects as unknown and assign a small weight to the regularizer in Eq. (2). Table 2 compares training with and without this term. On defects recorded by the annotators, including the term increases task-averaged Recall@3 from 41.6% to 42.8%. When each category with annotated positives receives equal weight, category-averaged Recall@3 rises from 23.8% to 24.8%.

AuthorsOpenReview ↗

Response to Reviewer qnAZ (Part 2/5)

Q1 — Can JACE guide the optimization of a video generator as a reward model?

Thank you for suggesting this direction. We designed JACE as a judge that identifies defects in generated videos and explains how those defects influence the choice between two candidates. Your suggestion extends this question to generation itself. Can JACE's scores serve as a reward signal for training a video generator?

When a judge provides feedback during training, the generator changes its outputs in response and may produce videos that are rare in the preference calibration data. Assessing whether the scores still agree with human preferences on these videos requires fine-grained preference annotations covering the corresponding generation behaviors. Given the computing resources available for this revision, we conducted a preliminary experiment with Wan2.1-1.3B. For each training prompt, Wan generated four candidate videos, and the frozen JACE judge scored each one. We used these scores as rewards and updated Wan's LoRA parameters with Flow-GRPO based on the relative scores within each group.

On prompts held out before training, we compared the original and fine-tuned Wan using the same random seed for each pair. In the main experiment, the fine-tuned model's JACE score increased by 1.104 on average (paired 95% CI [+0.732, +1.482]). JACE gave the fine-tuned video a higher score in approximately 68% of pairs. The VBench dimensions showed mixed changes. Dynamic Degree rose from 45.3% to 64.1%, a gain of 18.8 percentage points. Subject Consistency increased by 5.95%, and Imaging Quality showed no appreciable change. This preliminary experiment shows that Wan can use JACE's feedback to achieve higher rewards on unseen prompts. Future work can examine how reliably the reward reflects human preferences as the distribution of generated videos changes.


Q2 — Sharing defect weights across tasks

  1. Why we share weights across tasks. Defects such as blur and unnatural motion can occur in text-to-video generation, image-to-video generation, reference-guided generation, and video editing. Sharing weights lets us pool preference examples across tasks to estimate how much the quality score should change when the diagnostic score for a given defect changes by one standardized unit. Each task includes only the categories that apply to it. For example, poor reference consistency contributes to the score for tasks with a reference input and is excluded from the score for text-only generation.

  2. Why the same defect contributes differently across video pairs. Under Eq. (5), for a defect category applicable to both candidates, its contribution to the quality score of A relative to B is its weight multiplied by the difference between the standardized defect scores (B minus A). Consider blur. If A and B have the same blur score in one pair, blur contributes zero to the choice. If A's blur score exceeds B's by one standardized unit in another pair, blur lowers A's relative quality score by an amount equal to the blur weight. Both comparisons use the same weight. The magnitude and direction of the contribution depend on how the two candidates differ.

  3. Scope of the shared weights. Table 3 compares two schemes using the same diagnostic outputs. One applies a single penalty coefficient to all defects. The other learns a separate coefficient for each defect category and shares it across tasks. The latter raises GSB accuracy from 56.3% to 58.3%, supporting the use of category-specific weights. Across tasks, the same standardized score difference for the same applicable defect makes the same contribution to preference. For each video pair, JACE computes the contribution of each defect to the final choice using the categories applicable to the task and the diagnostic score differences between the two candidates.

AuthorsOpenReview ↗

Response to Reviewer qnAZ (Part 3/5)

Q3 — Precision of defect predictions and identification of videos with no salient defects

  1. Why we use Recall@3. Video annotations record the salient defects that raters observed. Our first question is how many of these defects appear among the model's top three predictions. For each video, Recall@3 (Appendix C.3) measures the fraction of recorded defects covered by the three highest-scoring applicable categories. This uses only confirmed defects and makes no assumption about whether unrecorded defects are present. When a video has no recorded defects, the denominator is zero, so we separately evaluate the model's ability to identify videos with no salient defects. The human review below assessed how many of the top three predicted defects were present.

  2. Determining whether a video contains a salient defect. We randomly sampled 25 videos from each of the four tasks in the attribution confirmation set, for a total of 100. Reviewers inspected the generation conditions, watched the videos with audio, and recorded a final judgment on whether each video contained a salient defect. They had no access to the original labels or model outputs. Taking videos judged by humans to have no salient defects (GOOD) as the positive class, we computed GOOD precision, recall, and F1 over all 100 videos.

  3. Checking individual defects in videos with a human-confirmed salient defect. For each video judged to contain a salient defect in the first step, we took JACE's three highest-scoring categories applicable to the task. We shuffled their order, hid the scores, and asked reviewers to judge whether each predicted defect was present. Precision@3 on this subset is the number of predictions confirmed by reviewers divided by the total number of predictions in the subset.

MethodDefect Precision@3¹ (%)GOOD Precision² (%)GOOD Recall² (%)GOOD F1² (%)
JACE63.472.788.980.0

¹ The 82 videos with salient defects yielded 246 predictions, three per video. Reviewers confirmed 156 of them.

² Of the 100 videos, 18 were labeled GOOD by the reviewers and 22 by the model. Both labeled the same 16 videos GOOD.


Q4 — Agreement between JACE's defect contributions and human reasons for pairwise preferences

We conducted a blinded evaluation on the held-out test set to examine whether JACE's defect contributions capture the reasons people give for preferring one video over another. We randomly sampled 25 video pairs from each of the four generation tasks. For each pair, three raters independently viewed the original conditioning input and the two candidate videos in random order. Each rater chose A, B, or a tie, then described in their own words the single difference that most influenced the decision. The raters had no access to the model predictions or the defect taxonomy. For example, “I prefer B because the motion transitions in A look unnatural” maps to a motion or transition defect favoring B. We rank defects by the absolute magnitude of their contribution to the score difference between A and B. The sign indicates which candidate the contribution favors. We count a top-k directional match when at least one of the top k contributions matches both the defect category in the human explanation and the candidate it favors.

For each pair, we constructed two rankings from the same defect predictions. The equal-weight baseline ranks defects by the absolute standardized score difference between the two videos. JACE multiplies each standardized difference by the corresponding category weight learned from a separate preference training set, then ranks the resulting contributions by absolute magnitude. We compared both rankings with the same human explanation to test whether weighting defects brings the raters' main reason closer to the top. We aggregated preference votes and defect attributions by video pair. After excluding ties, pairs with inconsistent judgments, and explanations that could not be mapped to a category, 82 pairs remained.

Basis for ranking defectsTop-1 directional matchTop-3 directional match
Largest standardized difference between the two videos (equal weights)22/82 (26.8%)43/82 (52.4%)
Largest contribution to the final score difference (JACE)28/82 (34.1%)55/82 (67.1%)

For 67.1% of the non-tied pairs with a mappable explanation, at least one of JACE's three highest-ranked contributions identified the defect cited by the raters and favored the same candidate. The corresponding rate was 52.4% with equal weights. Learning the weights improved the top-3 directional match rate by 14.6 percentage points.

Reviewer_fuC3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors propose JACE, an audio-video evaluator that combines defect attribution with human preference modeling. It first learns defect predictions from sparse annotations using omni-modal and auxiliary representations, then calibrates defect contributions using independent pairwise preferences. The resulting scores support video-level diagnosis, pairwise comparison, and generator ranking. Experiments across four generation tasks show improvements in defect detection, human preference agreement, and generator ranking, supported by ablation studies.

Critical strengths
  1. JACE learns defect detection and preference calibration from separate datasets, avoiding the need for both annotation types on the same videos.

  2. Effective defect attribution. The category-specific residual correction improves task-mean Recall@3 from 42.8% to 56.5%, supporting the use of complementary representations.

  3. Interpretable preference modeling. The scoring function explicitly decomposes quality differences into base quality and defect contributions. The ablations show that combining these components improves preference prediction.

  4. Strong empirical performance. JACE outperforms existing evaluators in defect Recall@3, pairwise preference accuracy, and generator ranking across four generation tasks.

Critical weaknesses
  1. Limited comparison with closely related methods. Attribution-based evaluators such as Q-Save and REACT are discussed but not included in the quantitative comparisons. This makes it harder to assess JACE's advantages over existing interpretable evaluation methods.

  2. Limited validation of attribution explanations. While JACE provides explicit defect contributions, their agreement with human reasoning is not evaluated. Figure 4 presents one qualitative example, leaving the reliability of these explanations unclear.

  3. Insufficient audio-video analysis. The results are mainly reported by generation task, without separate analysis of audio quality, synchronization, or cross-modal defects. This makes it difficult to assess JACE's effectiveness for joint audio-video evaluation.

  4. Limited gains from preference calibration. Learning defect-specific weights improves GSB accuracy from 56.3% to 58.3% over equal weighting, as shown in Table 3. The gain is relatively modest, and its statistical significance is not reported.

Questions for authors
  1. Could the authors include comparisons with Q-Save or REACT, or clarify why these methods are not evaluated?

  2. Could the authors validate whether the predicted defect contributions agree with human explanations of pairwise preferences?

  3. Could the authors report separate results for audio-related defects and audio-video synchronization, rather than only task-level averages?

  4. Are the improvements from learned defect weights statistically significant across different tasks? Per-task results or confidence intervals would help clarify this.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fuC3 (Part 1/2)

Dear Reviewer fuC3,

Thank you for acknowledging our contributions and your valuable comments. We address your concern as follows:


W1 / Q1 — Quantitative comparisons with Q-Save and REACT

When selecting attribution baselines for Table 1, we considered whether each method's outputs could be evaluated under a common protocol for video-level defect detection. Q-Save uses 3 broad categories, whereas JACE distinguishes 46 defect categories across 5 groups. The two taxonomies differ in both granularity and coverage. REACT focuses on frame-level structural distortions, with 8 types across 2 groups. A direct comparison in Table 1 would therefore depend on both model performance and the mapping between taxonomies. We did not define such a mapping in the original manuscript, and we could not verify the availability of official downloadable weights and runnable inference code for either baseline. We therefore did not report a direct attribution comparison.

Both papers report results on VideoGen-RewardBench, which provides a common benchmark for pairwise preferences in visual quality (VQ) and motion quality (MQ). The table gives JACE's accuracy under the benchmark protocol alongside the values reported in the Q-Save and REACT papers.

MethodVQ w/ TiesVQ w/o TiesMQ w/ TiesMQ w/o Ties
Q-Save40.3467.49——
REACT40.2053.8038.6062.60
JACE54.4174.6459.0672.18

Under both tie protocols, JACE achieves higher VQ accuracy than Q-Save and REACT and higher MQ accuracy than REACT. These results compare the methods' preference rankings on a common benchmark.


W2 / Q2 — Agreement between JACE's defect contributions and human reasons for pairwise preferences

We conducted a blinded evaluation on the held-out test set to examine whether JACE's defect contributions capture the reasons people give for preferring one video over another. We randomly sampled 25 video pairs from each of the four generation tasks. For each pair, three raters independently viewed the original conditioning input and the two candidate videos in random order. Each rater chose A, B, or a tie, then described in their own words the single difference that most influenced the decision. The raters had no access to the model predictions or the defect taxonomy. For example, “I prefer B because the motion transitions in A look unnatural” maps to a motion or transition defect favoring B. We rank defects by the absolute magnitude of their contribution to the score difference between A and B. The sign indicates which candidate the contribution favors. We count a top-k directional match when at least one of the top k contributions matches both the defect category in the human explanation and the candidate it favors.

For each pair, we constructed two rankings from the same defect predictions. The equal-weight baseline ranks defects by the absolute standardized score difference between the two videos. JACE multiplies each standardized difference by the corresponding category weight learned from a separate preference training set, then ranks the resulting contributions by absolute magnitude. We compared both rankings with the same human explanation to test whether weighting defects brings the raters' main reason closer to the top. We aggregated preference votes and defect attributions by video pair. After excluding ties, pairs with inconsistent judgments, and explanations that could not be mapped to a category, 82 pairs remained.

Basis for ranking defectsTop-1 directional matchTop-3 directional match
Largest standardized difference between the two videos (equal weights)22/82 (26.8%)43/82 (52.4%)
Largest contribution to the final score difference (JACE)28/82 (34.1%)55/82 (67.1%)

For 67.1% of the non-tied pairs with a mappable explanation, at least one of JACE's three highest-ranked contributions identified the defect cited by the raters and favored the same candidate. The corresponding rate was 52.4% with equal weights. Learning the weights improved the top-3 directional match rate by 14.6 percentage points.

AuthorsOpenReview ↗

Response to Reviewer fuC3 (Part 2/2)

W3 / Q3 — Detection of audio quality and audio–visual synchronization defects

Appendix A defines nine audio quality categories (D32–D40), including audio–visual desynchronization (D37) and mistimed sound effects or dialogue (D38). We report Recall@3 (%) averaged across the nine categories and separately for the two timing-related defects.

Defect categoryQwen3-OmniGemini 3.7 FlashJACE
Audio quality overall (category mean, D32–D40)16.8926.5643.95
Audio–visual desynchronization (D37)14.0630.8645.70
Mistimed sound effects or dialogue (D38)9.7720.3136.33

JACE achieves higher audio defect Recall@3 than both Qwen3-Omni and Gemini 3.7 Flash. On the two timing-related defects, JACE achieves 45.70% Recall@3 for audio–visual desynchronization and 36.33% for mistimed sound effects or dialogue. These gains over the stronger baseline, Gemini 3.7 Flash, are 14.84 and 16.02 percentage points, respectively.


W4 / Q4 — Gains from learning defect weights and their consistency across tasks

As shown in W2/Q2, learning defect weights increases the top-3 directional match rate from 52.4% to 67.1%, indicating closer agreement between JACE's ranking of defect contributions and the reasons raters give for their preferences.

In Table 3, both methods use the same quality head and defect predictions. Overall GSB accuracy rises from 56.3% with equal weights to 58.3% with JACE, a gain of approximately 2.0 percentage points. The table below gives the per-task GSB accuracies, their differences, and the 95% confidence intervals for those differences.

Generation taskEqual-weight GSB (%)JACE GSB (%)Difference (pp)95% CI for the difference (pp)
T2V57.060.2+3.13[+1.17, +5.47]
I2V55.557.0+1.56[+0.39, +3.13]
R2V53.153.9+0.78[−0.78, +2.34]
V2V59.862.1+2.34[+0.78, +4.30]
Overall56.358.3+1.95[+1.07, +2.83]

The overall gain is approximately 2.0 percentage points (95% CI [+1.07, +2.83]). T2V and V2V have the largest estimated gains, followed by I2V. The 95% intervals for all three tasks lie above zero. For R2V, the point estimate is positive, but the interval includes zero.

I appreciate authors' efforts on the rebuttal. After carefully reviewing the rebuttal and revisiting the paper, most of my concerns have been solved. Therefore, I'm willing to raise my score from 2 to 3.

Dear Reviewer,

Thank you for carefully reviewing our rebuttal and revisiting the manuscript. We appreciate your thoughtful reassessment and are glad that our responses have addressed most of your concerns.

Thanks again for your valuable input and support.

Best regards,

The Authors of Submission 59652

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author Response: Summary of Key Additions and Clarifications

Dear Area Chair and Reviewers,

Thank you for your careful reading of our paper. We appreciate the strengths each reviewer highlighted:

  • Reviewer fuC3 recognized the value of learning from separate defect and preference annotations, the interpretability of the scoring function, and JACE’s performance across four generation tasks.
  • Reviewer qnAZ highlighted the practical motivation for the two-stage design, which allows defect diagnosis and preference calibration to draw on independently collected data.
  • Reviewer hoQU appreciated how JACE connects individual defects to pairwise judgments and generator rankings, and how the controlled ablations support its design choices.

Our rebuttal brings together three main additions.

1. Connecting defect attribution to human judgment

Identifying what went wrong in a video and deciding which of two videos is preferable require different supervision. To test how well JACE connects the two, we asked viewers to explain their preferences in their own words. Among non-tied pairs with a consistent judgment and an explanation that could be mapped to a defect category, one of JACE’s top three contributions matched both the defect viewers cited and the candidate they favored in 67.1% of pairs, compared with 52.4% under equal weighting. Learning the weights thus helped bring the defects viewers cited closer to the top of the ranking.

2. Establishing what the diagnostic results tell us

We supplemented retrieval of annotated defects with a blinded audit. Reviewers checked whether the defects JACE flagged were visible in the videos and how often JACE flagged videos they judged free of salient defects. We also describe how annotators resolved disagreements and assigned ties, and report audio quality and synchronization separately so readers can assess these capabilities alongside visual diagnosis.

3. Broadening the evidence for where JACE applies

VideoGen-RewardBench provides a common basis for comparison with related evaluators. JACE achieved higher visual-quality preference accuracy than the published results for Q-Save and REACT, and higher motion-quality preference accuracy than REACT. We also examined newer generator versions and alternative diagnostic backbones. A preliminary experiment with Wan showed that fine-tuning could increase JACE’s reward on held-out prompts.

Our responses to the individual reviewers provide the experimental protocols and detailed results for these analyses. Thank you again for your thoughtful feedback.

Best regards,
The Authors of Submission 59652

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewers,

Thank you again for your time and thoughtful feedback on our manuscript.

As the discussion period is nearing its end, we would be grateful if you could share your feedback on our responses and revised manuscript at your earliest convenience.

Best regards,

The Authors of Submission 59652

Dear Area Chair and Reviewers,

Thank you again for your time and thoughtful feedback on our manuscript. We appreciate the follow-up feedback received so far.

As the discussion period is nearing its end, we would be grateful if you could share any further feedback on our responses and revised manuscript at your earliest convenience.

Best regards,

The Authors of Submission 59652

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.