acceptodds
Under review as a conference paper at ICLR 2027

Vibrics: Unified Evaluation of Video Generation Pipelines with Request-Specific Rubrics

Abstract

Video generation pipelines rewrite open-ended user requests into model-ready prompts before rendering videos. Existing evaluators focus on final outputs, providing little evidence about rewriting decisions and often missing fine-grained, request-specific requirements. We introduce Vibrics, a unified evaluation framework that freezes a request-specific rubric derived from the original request and reference media before observing any candidate. Its criteria evaluate the rewritten prompt, any available process trace, and the delivered video under one shared standard. Video criterion decisions yield Fulfillment (request satisfaction), Craft (visual quality), and calibrated Utility, while the full criterion record provides evidence-grounded diagnoses of unmet requirements across the pipeline. To benchmark complete pipelines at scale, we build Vibrics-Bench from 240 human-reviewed, open-ended production requests that reflect what users would type or paste, span eight input types and professional scenarios, and require substantive rewriting. Crossing five rewriting methods with five generators yields 25 pipelines per request and 6,000 videos, all evaluated under one frozen rubric per request. Vibrics agrees with expert consensus on 85.3% of comparable criterion decisions, while its Fulfillment, Craft, and Utility rankings reach 81%/81%/78% of human-human agreement. Experts also find that 89.6% of 1,000 diagnostic reports identify the delivered video's most important problem. On the 40-request annotated subset, Craft varies primarily with the generator, whereas Fulfillment varies with both the generator and the rewriting method. Together, Vibrics provides comparable scores and diagnoses for complete pipelines, while Vibrics-Bench provides a shared benchmark for evaluating rewriters, generators, and their combinations.

Primary area:
datasets and benchmarks
Keywords:
video generation evaluation, rubric-based evaluation, multimodal LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a19TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Vibrics evaluates video generation pipelines using rubrics from the original request. It measures request fulfillment and video quality, and gives diagnostic reports. The benchmark contains 6,000 videos from five rewriters and five generators.

Critical strengths
  1. Using the original request is useful. It can find requirements that are lost during prompt rewriting.
  2. The crossed design helps compare rewriters and generators. The paper also includes human checks and separate calibration data.
Critical weaknesses
  1. The score can exclude requirements that were not checked. In C01, guzheng accompaniment is N/A, but Fulfillment is 1.000. It is unclear whether this requirement was verified elsewhere. Tables 17 and 18 also disagree: under the stated equal weighting, the Seedance–LTX Fulfillment gap is 0.1820 from Table 17, but 0.174 in Table 18.
  2. Most text-conditioned baselines receive the rewritten prompt, while Vibrics uses the original request. This can favor Vibrics. VideoArgus is an exception, but its rubric and judge settings are unclear. Also, the 89.6% diagnostic result only checks the main video problem. It does not show that Vibrics correctly assigns the problem to a pipeline stage.
Questions for authors
  1. Why is guzheng accompaniment N/A in C01 when Fulfillment is 1.000? How do unchecked requirements affect the score?
  2. Why do Tables 17 and 18 give different contrasts? Do they use the same data and weights?
  3. How do the results change if baselines receive the same original request and references?
  4. Was stage-level diagnosis checked by human annotators, beyond finding the main video problem?
4 comments
AuthorsOpenReview ↗

Response to Reviewer a19T

Thank you for closely checking the C01 example, the aggregation results, and the baseline evaluation protocol. We corrected the affected audio judgment and ranking tables, reran input-controlled comparisons, and added expert checks of the judgments used to diagnose failures at different stages.

W1 & Q1 & Q2

The C01 example marks explicitly requested guzheng audio as N/A despite a perfect Fulfillment score, and the Seedance–LTX differences are inconsistent across two tables. What caused these errors, and how were they corrected?

Response

We rechecked C01 and found that the Omni audio-evaluator call had failed because of an input-formatting error. The failure to obtain a judgment did not make the requested audio inapplicable. Inspection of the delivered video confirmed that it contains no audio track, so the explicit guzheng-accompaniment requirement must be marked FAIL rather than N/A. Including this applicable criterion changes Fulfillment from 1.00 to 37/38 = 0.97; Craft remains 0.68, and uncalibrated Utility becomes 0.86. We also reviewed historical audio-evaluator failures and propagated verified corrections through the affected scores and rankings.

For the inconsistent Seedance–LTX comparison, the original tables used different effective sets of valid records and different aggregation orders: one summarized pipeline-combination means, while the other averaged within-request contrasts. We recalculated both from the same valid request-level records, giving each valid request equal weight within a combination and each rewriting method equal weight when comparing generators. The contrast now comes from the unrounded combination means, producing a Seedance–LTX Fulfillment difference of 0.18 in both summaries after rounding. The leading rewriter and generator remain unchanged after the corrections.

The revised results now apply one consistent rule for criterion applicability and one consistent aggregation protocol across the case analysis and benchmark comparisons. Appendices A.2 and F.1 (Table 28) document the corrected C01 case; Appendix D (Tables 22–23) reports the corrected matrix and request-clustered intervals.

W2 & Q3 & Q4

Are external evaluators compared with suitable access to the original request and references? Does the 89.6% diagnostic result also validate judgments about where a requirement was lost?

Response

We reran five text-conditioned evaluators on the same 40 requests and 1,000 human-rated videos, giving each either the original request or the rewritten generation prompt. The table reports request-averaged Kendall's τ\tau-b against expert overall ratings:

EvaluatorRewritten prompt + videoOriginal request + video
CLIPScore0.010.08
VQAScore−0.050.02
VideoScore−0.12−0.13
VBench Semantic−0.050.04
Qwen holistic judge0.330.34
VideoArgus (its own request-derived rubric)—0.37
Vibrics U0U_0 (request-derived rubric)—0.43

Four of the five text-conditioned baselines improve with the original request. VideoArgus constructs its own request-specific rubric, while Vibrics derives its rubric from the original request and any references provided. We report the actual reference access of each method. More directly, our matched agent comparison supplies both free-form and Vibrics-guided agents with the same original request, references, candidate video, backbone, and tools. With Flash, overall-preference ordering accuracy improves from 61.62% to 71.77% when the agent uses Vibrics's shared criteria and evidence. This shows the benefit of the evaluation procedure beyond simply changing the source text.

We also clarified the two forms of diagnostic validation. The reported 89.6% measures identification of the most important problem in the final video. In a separate two-expert audit, the criterion judgments supporting stage-level diagnosis agree with expert consensus at 87.23% for rewritten prompts, 80.00% for available process traces, and 84.62% for delivered videos. These results validate both the evaluator's ability to identify the main problem and the stage-specific evidence used to explain it.

The corrected input comparison and agent results appear in Section 4.1 and Table 3; Section 4.2 and Appendices C.1 and C.5 document the distinct diagnostic checks and their agreement with experts.

Dear Reviewers,

We would like to kindly and respectfully follow up, as the rebuttal period is drawing to a close. In addition to our response to the reviews, we have also submitted a revised version of the paper along with a summary of the changes. We have carefully addressed all raised concerns, and we would greatly appreciate your feedback whenever you have the opportunity.

Thank you very much for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Reviewer Feedback Before the Discussion Deadline

Dear Reviewer a19T,

As the discussion deadline is approaching, we understand that you may be very busy. If you could spare some time to share your thoughts, we would sincerely appreciate it and would be very happy to discuss any remaining questions in greater detail.

We are encouraged that Reviewer AtcM has kindly followed up and reaffirmed their positive assessment of our paper. We would also greatly value your feedback on whether our responses, additional experiments, and revisions have adequately addressed your concerns.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_a19TOpenReview ↗

Response

Some of my concerns have been addressed, so I am willing to keep my original score. I still have reservations regarding the level of innovation, but if the paper is accepted, I have no objection.

Reviewer_VcUNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VIBRICS, an evaluation framework and benchmark that uses request-specific rubrics to assess video generation pipelines in terms of requirement fulfillment and visual quality, while diagnosing failures across prompt rewriting and video generation.

Critical strengths

The paper is well written.

Critical weaknesses
  1. The evaluation should distinguish the capabilities of the underlying models from the effectiveness of the rewriting strategies. However, the compared rewriters differ in both their base models and their workflows, making it difficult to attribute performance differences to either factor. A more controlled design would compare rewriting strategies using the same base model, and separately compare base models using the same rewriting strategy.

  2. I am not convinced that automatically constructing evaluation criteria provides substantial benefits over directly using a capable general-purpose agent such as Codex. Given a generation prompt and the corresponding video, such an agent could plausibly provide a comprehensive assessment of instruction compliance, visual artifacts, and other quality issues. The authors should demonstrate clear improvements over a strong direct-evaluation baseline with access to the same original request and reference materials. Without this comparison, the need for a dedicated rubric-construction framework remains insufficiently justified.

  3. Video generation is expensive, and the benchmark requires 25 generations per request to compare different rewriter–generator combinations. Although this is an experimental configuration rather than a requirement for producing a single video, it entails substantial benchmarking costs. More importantly, the paper does not adequately quantify the cost and latency of rubric construction and per-video evaluation. Without these measurements, the framework’s practicality in real production workflows remains unclear.

  4. The underlying evaluators still lack sufficiently reliable alignment with human aesthetic preferences, which limits the trustworthiness of their quality assessments. Automatically generating additional criteria does not, by itself, resolve this problem. I would consider improving the alignment of open-source evaluators with human aesthetic judgments a more valuable research direction. In its current form, the paper does not convincingly establish that its criterion-construction framework offers meaningful advantages beyond what existing general-purpose agents can already provide.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer VcUN [1/2]

Thank you for examining what Vibrics contributes beyond a capable general-purpose agent and how meaningfully the benchmark compares different rewriting methods. We clarified the distinction between complete-pipeline comparisons and controlled strategy experiments, and added matched agent evaluations, repeated-run measurements, visual-quality comparisons, and evaluation-cost analysis. We address your four concerns below.

W1 & Q1

The rewriters differ in both underlying models and rewriting strategies. How can the benchmark distinguish complete-system performance from the effect of the rewriting strategy itself?

Response

The original five-rewriter × five-generator experiment compares complete rewrite–generate pipelines under one evaluation standard. Each entry combines a rewriting system and a video generator, evaluated against the same original request. This supports fair comparisons of the delivered pipelines without treating the resulting rankings as a causal ranking of rewriting algorithms alone.

We conducted a controlled experiment that fixes the rewriting backbone, video generator, original requests, references, and Judge configuration, changing only the rewriting method. With Qwen3.7-Max as the rewriting backbone, Wan 2.7 as the generator, and ten requests, the results are:

Same-backbone rewriting comparisonFulfillmentCraftUncalibrated U0U_0
APE0.900.630.80
Structured Skill0.760.650.72

Under these matched conditions, APE achieves 0.14 higher Fulfillment and 0.08 higher U0U_0, while Structured Skill has slightly higher Craft. The comparison distinguishes the effect on preserving user requirements from the effect on visual quality.

These two experiments establish complementary uses of Vibrics-Bench: comparing deployed pipelines as complete systems and evaluating individual rewriting strategies under controlled conditions, always against the same original request. We clarify this distinction in the main discussion and report the controlled comparison in Appendix E.1 (Table 27).

W2

A strong general-purpose agent can already inspect a video and assess its quality. What additional value does Vibrics provide when both approaches receive the original request, references, and video?

Response

A capable general-purpose agent can identify video artifacts and assess whether a video follows a prompt. Vibrics addresses a further requirement: different rewrite–generate pipelines must be evaluated against the same original user request, even when their rewritten prompts omit different requirements. If a candidate is judged only against its own rewritten prompt, a requirement lost during rewriting may also disappear from the evaluation standard.

For example, suppose the original request asks for a particular label to appear in the video. A rewriter may omit that label, and the generator may faithfully follow the incomplete rewrite. Evaluating only the generated video against that rewrite can miss the original omission. Vibrics instead derives a set of criteria from the original request and references before inspecting the candidates, freezes those criteria, and applies them to the rewritten prompt, any available process evidence, and the final video. The same requirement can therefore be checked at different stages and compared consistently across pipelines. This is a shared evaluation procedure with traceable evidence, not merely an alternative wording of a video-judging prompt.

Crucially, our strong free-form agent baseline also receives the original request and references. We compare its evaluation with Vibrics-guided evaluation while keeping the backbone, candidate videos, source information, and available tools fixed:

Matched agent evaluationOverall-preference ordering accuracy
Flash, free-form61.62%
Flash, Vibrics-guided71.77%
Max, free-form56.96%
Max, Vibrics-guided67.98%

With Flash, Kendall's τ\tau-b also improves from 0.40 to 0.45. We examined repeatability on five videos, each evaluated three times: the average within-video standard deviation on a common 0–10 score scale decreases from 0.48 for free-form evaluation to 0.13 for Vibrics's criterion aggregation. This test illustrates how a fixed criterion-based procedure reduces variation across repeated assessments.

The controlled results demonstrate that Vibrics improves agreement with human preferences and produces more repeatable scores while keeping the evaluation target consistent across complete pipelines. Its retained criterion-level records also show whether a requirement was omitted in the rewrite or remained unmet in the video. We added the matched comparison to Section 4.2 and Table 3 and the repeatability analysis to Appendix C.5 (Table 20); Appendix G.1 reports evaluation resources.

AuthorsOpenReview ↗

Response to Reviewer VcUN [2/2]

W3

How costly is Vibrics in practice? In particular, how do the 25 video generations used by the benchmark relate to the time and tokens required to construct a rubric and evaluate an individual video?

Response

The 25 generated videos per request are part of the benchmark's crossed comparison, not a requirement for evaluating a new video. Vibrics creates the request-specific rubric once and reuses it whenever another candidate video is evaluated against that request.

In our recorded timing measurements, rubric construction averages approximately 21.5 minutes per request, while complete evaluation averages approximately 27.1 minutes per candidate video. The two phases are measured separately because the construction cost is shared by all candidates for the request. Appendix G.1 additionally reports provider-recorded token usage: 281.6K input / 140.8K output tokens for construction, and 1,047.7K / 286.9K for complete candidate evaluation. Its measured 147.9-second parallel criterion-judging time is a distinct component of the evaluation process, not the complete per-video runtime.

Reusing one frozen rubric and evaluating candidate videos independently makes the cost structure explicit and allows the same standard to support both benchmark-scale comparisons and evaluation of new candidates. Section 3.2 describes rubric reuse; Appendix G.1 (Table 33) reports token usage and the parallel judging measurement.

W4

How well do the evaluators agree with human aesthetic preferences? Does structured evaluation offer a meaningful improvement over a strong general-purpose agent on visual quality itself?

Response

We agree that improving agreement with human aesthetic preferences is an important research direction, and we have made substantial efforts to investigate this problem in Vibrics. Our framework makes visual-quality criteria explicit, directs evaluators to relevant video evidence, and reports Craft separately from Fulfillment. This allows us to study how well evaluators assess visual quality without conflating it with whether the original user requirements are satisfied.

Importantly, we conducted a dedicated human evaluation involving two independent experts and 1,000 videos across 40 requests and 25 rewrite–generate pipeline combinations. Both experts assessed the videos against the original requests and references, providing separate ratings for visual quality, requirement fulfillment, and overall satisfaction. These annotations allow us to evaluate aesthetic judgments directly against human preferences rather than relying solely on automated evaluation metrics.

We further tested whether Vibrics improves visual-quality judgments using the same backbone, candidate videos, source inputs, and available tools for free-form and Vibrics-guided agents:

Visual-quality ordering accuracyFree-form agentVibrics-guided agent
Flash54.36%61.31%
Max48.90%58.95%

Vibrics guidance improves visual-quality ordering accuracy by 6.95 percentage points with Flash and 10.05 points with Max. The separately aggregated Craft score reaches 59.46% under Flash. For overall human preference, Flash improves from 61.62% to 71.77% with the Vibrics criteria and evidence.

Together, our dedicated human annotations and controlled experiments demonstrate that Vibrics meaningfully improves agreement with human visual-quality preferences using the same capable evaluator. Beyond assessing whether a video fulfills the original request, Vibrics provides an effective approach to evaluating its visual quality against human judgments. These findings establish the value of explicit evaluation criteria and evidence-based assessment for both dimensions of video-generation quality. We will continue investing in this direction by expanding our human-annotated evaluation set, involving more expert annotators, and further studying how to improve alignment with human aesthetic preferences. Sections 3.3 and 4.1–4.2 and Appendix C.5 report the human evaluation protocol and detailed results.

Dear Reviewers,

We would like to kindly and respectfully follow up, as the rebuttal period is drawing to a close. In addition to our response to the reviews, we have also submitted a revised version of the paper along with a summary of the changes. We have carefully addressed all raised concerns, and we would greatly appreciate your feedback whenever you have the opportunity.

Thank you very much for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Reviewer Feedback Before the Discussion Deadline

Dear Reviewer VcUN,

As the discussion deadline is approaching, we understand that you may be very busy. If you could spare some time to share your thoughts, we would sincerely appreciate it and would be very happy to discuss any remaining questions in greater detail.

We are encouraged that Reviewer AtcM has kindly followed up and reaffirmed their positive assessment of our paper. We would also greatly value your feedback on whether our responses, additional experiments, and revisions have adequately addressed your concerns.

Thank you very much for your time and consideration.

Best regards,
The Authors

Reviewer_AtcMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Vibrics, a unified evaluation framework for rewrite-then-generate video pipelines. For each original user request, Vibrics constructs and freezes a request-specific rubric before observing any candidate, decomposes the request into independently judgeable criteria, assigns each criterion to the rewritten prompt, process trace, or delivered video, and uses a Judge Agent to locate evidence and route it to a matched evaluator. It reports Fulfillment, Craft, calibrated Utility, and a diagnostic report. The authors build Vibrics-Bench with 240 human-reviewed production requests, crossing five rewriters and five generators to form 25 pipelines and 6,000 videos, and validate the framework on a 40-request, 1,000-video subset with expert annotations.

Critical strengths
  • The paper addresses an important evaluation gap: most existing video evaluators judge only the final video against a rewritten prompt and cannot localize where a user requirement was lost across the full pipeline. 
  • The frozen request-specific rubric is a sound design choice: it gives every pipeline the same standard and avoids treating different rewritten prompts as different evaluation tasks. 
  • The validation is multi-level, including rubric warrant audit, criterion-level agreement with experts, score-level human ranking agreement, and diagnostic-report evaluation. 
  • The benchmark scale and retained criterion-level evidence are useful for both system comparison and failure diagnosis.
Critical weaknesses
  • The framework is highly complex and depends on multiple LLM/VLM components for rubric construction, fusion, refinement, judging, and routing; reproducibility and robustness to component changes are not sufficiently demonstrated. 
  • Human validation is still limited: score-level validation uses only 40 requests, 1,000 videos, and two experts, while calibration uses only 20 requests; this weakens claims about cross-domain generality and statistical reliability. 
  • External evaluator comparisons are not fully input-matched, since some baselines see only prompt+video or video-only rather than the request-specific rubric, so comparative conclusions should be stated more cautiously.
Questions for authors
  • How stable are the rubric construction and Judge decisions under different backbone models, rubric writers, and evaluator configurations? 

  • Can the authors report confidence intervals or variance across different calibration splits for calibrated Utility? 

  • Does the 89.6% diagnostic-report accuracy measure only identification of the most important problem, or does it also verify correct attribution to prompt, process, or video? 

  • Can the framework be scaled to more annotated requests without prohibitive human or computational cost?

  • Since the framework heavily relies on specific LLM/VLM components (e.g., Qwen series) for rubric construction, judging, and routing, do the authors plan to conduct a systematic ablation study by swapping the backbone models?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AtcM [1/3]

Thank you for examining the robustness of Vibrics and the reliability of its human evaluation. We added component-replacement tests, calibration-split analyses, and human checks of the judgments used in stage-level diagnosis.

W1 & Q1 & Q5

Vibrics uses several model components for rubric construction and evaluation. How stable are its judgments when the rubric writer, Judge, or visual evaluator is replaced, and can these configurations be reproduced?

Response

We tested the effects of replacing model components at three stages: rubric construction, the Judge, and the visual evaluator. The experiments use videos associated with ten requests. In the first comparison, A and B share the same frozen rubric and change only the Judge; B gathers its own evidence rather than reusing A's judgments. B and C use the same Judge but change rubric construction, including fusion and refinement.

ConfigurationRubric construction, fusion, and refinementJudge backbone
AMaxMax
BSame frozen rubrics as AFlash
CFlashFlash
Ordering accuracy against human preferencesABC
Fulfillment56.60%59.00%58.05%
Visual quality60.10%60.04%60.52%
Overall preference73.47%72.78%70.94%

Replacing only the Judge changes overall-preference accuracy by 0.69 percentage points; replacing rubric construction with the Judge fixed changes it by 1.84 points. The evaluation criteria remain common to all candidates under each configuration, so backbone replacement does not give different candidates different requirements.

We also held the rubric, judgment prompts, and ten-frame video-sampling protocol fixed while replacing the visual evaluator; audio and program checks were unchanged:

Visual evaluatorFulfillment ordering accuracyVisual-quality ordering accuracyOverall-preference ordering accuracy
Qwen3-VL-32B-Instruct70.10%57.66%73.06%
Qwen3-VL-Plus73.82%58.73%74.40%
DeepSeek-v4.1-Flash75.64%64.13%77.26%

Across these tests, the framework retains strong human-preference agreement with different component choices, including a visual evaluator outside the Qwen family. Appendix B.4 and Tables 11–12 now document the A/B/C protocol, independent evidence collection, replacement settings, and all three evaluation targets. We also specify the frozen rubrics, prompts, aggregation rules, calibration map, and criterion-level evidence in the reproducibility materials.

W2

The human-evaluation sample and calibration set are relatively small. More evidence is needed to support the reliability of the scores, diagnostic judgments, and agreement with human preferences.

Response

Our human evaluation covers 40 requests, all 25 pipeline combinations, and 1,000 videos. Two independent experts assess request fulfillment, visual quality, and overall satisfaction. A disjoint set of 20 requests is used only to calibrate the score scale. We additionally evaluated sensitivity across ten calibration/evaluation splits, obtaining a positive mean Kendall improvement in every split.

The validation goes beyond overall scores. Experts confirm that over 97% of audited criteria are grounded in the input. Vibrics agrees with expert consensus on 85.3% of comparable criterion judgments, and 89.6% of the reports identify the video’s primary problem. A separate human audit also supports the judgments based on the prompt, process, and video evidence used in the diagnostic reports, with agreement around 80% or higher across stages. Together, these expert checks establish a basis for both the scores and the stage evidence used to explain missing requirements.

We have also strengthened the most relevant human-alignment comparison. With the same Flash backbone and candidate videos, providing Vibrics’s criteria and evidence improves ordering accuracy from 61.62% to 71.77% and Kendall agreement from 0.40 to 0.45. The crossed design, expert validation, calibration analysis, and controlled agent comparison provide complementary evidence for the reliability of Vibrics. Details appear in Sections 3.3–4.2 and Appendices C.1–C.5.

AuthorsOpenReview ↗

Response to Reviewer AtcM [2/3]

W3

The original external-evaluator comparison uses different inputs for different methods. Please clarify the information each baseline receives and compare methods on matched videos and request text.

Response

Vibrics constructs its request-specific rubric from the original request and reference media, freezes it before candidate evaluation, and applies it across candidate pipelines. The rubric is therefore an output of the evaluation procedure, rather than an additional human-written answer. We clarify each baseline’s inputs and extend the comparison to examine the effect of text-input choice.

CLIPScore, VQAScore, VideoScore, VBench Semantic, and the Qwen holistic judge accept text and video. We evaluate each using both the original request and the rewritten generation prompt on the same videos from the 40-request, 1,000-video human-evaluation cohort. We also include VideoArgus, which constructs its own request-specific rubric, and our original Vibrics U0 results. All entries below use the same videos and report Kendall’s τ\tau-b against human overall ratings, averaged first across experts and then across requests.

EvaluatorOriginal request + videoRewritten prompt + video
CLIPScore0.080.01
VQAScore0.02−0.05
VideoScore−0.13−0.12
VBench Semantic0.04−0.05
Qwen holistic judge0.340.33
VideoArgus0.37—
Vibrics U00.43—

For VideoArgus and Vibrics, the original-request setting includes the rubric constructed by each method; Vibrics also uses the reference media when provided. Neither is evaluated using the rewritten prompt as its source of requirements. VBench Quality is the only video-only baseline and shows near-zero Kendall agreement. CLIPScore uses the same ViT-L/14-336 checkpoint across candidates; VBench Semantic reports overall consistency, and VBench Quality follows the official seven-dimension normalization and weighting.

Using the original request improves agreement for four of the five text-conditioned evaluators. Vibrics U0 remains higher than these baselines under either text-input setting and also exceeds the rubric-based VideoArgus comparator. In our additional agent comparison in W2, both methods receive the same original request, references, video, backbone, and available tools. Section 4.1 and Table 3 now report both text-input settings. Appendix C.5 specifies rubric construction, reference access, VideoArgus implementation, and the updated CLIPScore, VideoScore, and VBench settings.

Q2

How much does calibrated Utility vary across different calibration/evaluation splits?

Response

We evaluate sensitivity to calibration splits using the 60 requests from the paper experiments. We generate 10 random splits, each assigning 20 requests to calibration and the remaining 40 to evaluation. For each split, the mapping is fitted using only the calibration requests, and uncalibrated U0 and calibrated Utility are compared on the same evaluation requests.

We use the 20 pipeline combinations available across all 60 requests to keep pipeline coverage consistent. Agreement with human overall ratings is measured by Kendall’s τ\tau-b, calculated separately for each expert within each request and then averaged across experts and requests.

MeasureMeanStandard deviationMinimumMaximum
Uncalibrated U0U_043.991.1742.5245.79
Calibrated Utility45.181.2443.4647.10
Within-split improvement+1.180.53+0.51+2.21

Calibrated Utility improves upon U0U_0 in all ten splits by an average of 1.18 Kendall points (100 ×\times τ\tau-b). These results show that the calibration benefit persists across the tested calibration/evaluation partitions. The “Stability across calibration samples and splits” paragraph in Appendix C.3 and Table 19 report the ten 20/40 request splits, their means and standard deviations, and paired calibration gains.

Q3

Does the reported 89.6% diagnostic accuracy measure only the main problem visible in the video, or also the attribution of failures to the prompt, process, and video stages?

Response

The two measurements evaluate different aspects of diagnosis. The previously reported 89.6% measures whether the diagnostic report identifies the main problem visible in the final video. It should not be read as an end-to-end causal attribution accuracy.

We also conducted a separate two-expert check of the judgments associated with the prompt, process, and video stages. Agreement with the experts was around 80% or higher across the three stages. These expert checks support the evidence used to explain whether a requirement was missing from the rewrite, the recorded process, or the final video. We have clarified the two evaluation targets in Section 4.2 and summarize the validation scope in Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer AtcM [3/3]

Q4

Can Vibrics be expanded to more annotated requests without requiring a separate, expensive rubric and human review for every candidate video?

Response

Vibrics constructs and reviews a request-specific rubric once, then reuses that frozen standard across candidate videos. Candidate evaluations are independent and can be run in parallel; the 25-video crossed benchmark is not required for evaluating a new video. This separates the human effort needed to validate a request's criteria from the automated assessment of additional candidates.

The benchmark already covers 240 requests. We also measured resource use across five cases: rubric construction averaged 281.6K input / 140.8K output tokens per request, while a full candidate evaluation averaged 1,047.7K / 286.9K tokens per video. Section 3.2 and Appendix G.1 (Table 35) now explain rubric reuse and report these costs separately.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Experimental Results [1/2]

Dear Area Chair and Reviewers,

Thank you for your time and thoughtful feedback. We appreciate the reviewers’ recognition of the value of evaluating complete pipelines against the original user request, the shared evaluation standard provided by frozen rubrics, and the benchmark’s human validation and crossed experimental design.

We have revised the paper to clarify the evaluation protocol and incorporate the additional evidence from our rebuttal. Key revisions are highlighted in blue, and the additional experiments and analyses are documented in the appendix.


1. Matched agent comparisons demonstrate improved prediction of human preferences.

Our human study provides independent ratings from two experts for 1,000 videos across 40 requests, covering requirement fulfillment, visual quality, and overall satisfaction. Using shared candidate videos, we compare free-form judging with Vibrics-guided overall judgments (Agent-U), keeping the backbone, original requests, reference media, and available tools fixed.

BackboneEvaluation methodOverall-preference ordering accuracy (%)Visual-quality ordering accuracy (%)
FlashFree-form agent61.6254.36
FlashVibrics-guided agent71.7761.31
MaxFree-form agent56.9648.90
MaxVibrics-guided agent67.9858.95

Vibrics guidance improves overall-preference ordering accuracy by 10.15 points with Flash and 11.02 points with Max. Visual-quality ordering accuracy also improves by 6.95 and 10.05 points, respectively. With Flash, Kendall’s τb\tau_b for overall preference increases from 0.40 to 0.45. These results demonstrate the value of explicit criteria and supporting evidence for assessing candidate videos with the same evaluator and source information (Sections 4.1–4.2, Table 3; Appendix C.5, Tables 19 and 21).

We also reran five text-conditioned baselines using both the original request and the rewritten generation prompt. The revised comparison reports both settings and documents each evaluator’s access to references and its rubric-construction procedure (Table 3; Appendix C.5).


2. Controlled rewriting experiments distinguish improvements in requirement fulfillment from visual quality.

We added a controlled comparison using the same Qwen3.7-Max rewriting backbone, Wan 2.7 generator, ten requests and source materials, frozen rubrics, and Judge configuration. Only the rewriting strategy changes.

Rewriting strategyFulfillmentCraftUncalibrated Utility U0U_0
Structured Skill0.760.650.72
APE0.900.630.80

Under these conditions, APE improves Fulfillment by 0.14 and uncalibrated Utility by 0.08, while Craft remains similar. The experiment shows that rewriting can improve satisfaction of the original request without a corresponding improvement in visual quality, supporting the separate measurement of Fulfillment and Craft.

The revised paper clarifies how Vibrics-Bench supports both comparisons of complete rewrite–generate pipelines and controlled comparisons of individual rewriting strategies. In both settings, candidates are evaluated under the same rubric, derived from the original request and references and frozen before observing any scored candidate (Sections 2.1 and 4.4; Appendix E.1, Table 27).

AuthorsOpenReview ↗

Summary of Revisions and Additional Experimental Results [2/2]

3. Rubric-based judging produces more stable repeated scores and remains effective across tested model configurations.

We examined the stability of repeated evaluations of the same videos. On a common 0–10 scale, free-form judging yields an average within-video standard deviation of 0.48, while our Judge following a fixed rubric and aggregating criterion-level decisions into U0U_0 reduces it to 0.13. This provides direct evidence that rubric-based judging produces more consistent scores across repeated evaluations, complementing the improvements in human-preference ordering above (Appendix C.5, Table 20).

We also tested rubric-construction and Judge substitutions on ten requests, keeping audio processing and program checks fixed.

Rubric constructionJudgeOverall-preference ordering accuracy (%)
MaxMax73.47
Same frozen Max rubricsFlash72.78
FlashFlash70.94

Replacing only the Judge changes ordering accuracy by 0.69 points. With the Judge fixed, replacing rubric construction changes it by 1.84 points. Additional visual-evaluator substitutions, including a non-Qwen model, achieve overall-preference ordering accuracies of 73.06%, 74.40%, and 77.26%, supporting the effectiveness of the framework across the tested component choices (Appendix B.4, Tables 11–12).

Calibration was evaluated separately across ten splits, each using 20 calibration requests and 40 evaluation requests. It improves Kendall agreement in every split, with a mean paired gain of 1.18 Kendall points on the 100×τb100\times\tau_b scale (Appendix C.3, Table 17).

These analyses are complemented by direct human validation. In a separate audit of 1,032 criterion instances, the two experts found 97.4% and 97.6% of the criteria supported by the original inputs, and Judge decisions agreed with expert consensus in 85.3% of comparable cases. Experts also found that 89.6% of diagnostic reports identified the main problem in the final video. Additional checks of prompt, process, and video criterion judgments achieved approximately 80% or higher agreement across stages, validating the evidence used in those reports (Sections 3.3 and 4.2; Appendices C.1–C.2).


Summary of revisions

  • Abstract, Section 1, Sections 4.1–4.2, Table 3, and Appendix C.5, Tables 19–21: Clarified the contribution of evaluating candidates against a shared standard derived from the original request. Added comparisons controlling the text inputs, matched general-purpose agent evaluations, visual-quality comparisons, and repeated-scoring evidence demonstrating the stability of rubric-based judging (Reviewers AtcM, VcUN, a19T).

  • Section 4.4 and Appendix E.1, Table 27: Distinguished complete-pipeline comparisons from the effects of individual rewriting strategies and added the controlled rewriting experiment above (Reviewer VcUN).

  • Appendix B.4, Tables 11–12, and Appendix C.3, Table 17: Added rubric-construction, Judge, and visual-evaluator substitutions, together with calibration sensitivity across ten request splits (Reviewer AtcM).

  • Sections 2.3, 3.3, and 4.2, and Appendices C.1–C.2: Clarified the human-validation protocol and the distinction between main-problem identification and stage-specific criterion judgments. Added expert checks of the judgments supporting diagnosis. We also clarify that video-stage criteria determine the scores, while prompt and process judgments provide diagnostic evidence (Reviewers AtcM, a19T).

  • Section 3.2 and Appendix G.1, Table 33: Clarified that the 25-pipeline matrix is part of benchmark construction and that new candidates reuse the request’s frozen rubric. Separately reported token consumption for rubric construction and complete video evaluation, together with the measured parallel criterion-judging time (Reviewers AtcM, VcUN).

  • Appendices A.2 and F.1, Table 28, and Appendix D, Tables 22–23: Corrected the missing requested audio in C01 to FAIL, giving Fulfillment of 37/38, and recalculated affected results using consistent valid records and aggregation rules. The leading rewriter and generator remain unchanged after these corrections (Reviewer a19T).

  • Appendices A.5–A.6, B.1–B.3, and G, Table 32: Documented construction and evaluation prompts, model and tool configurations, versioned manifests, criterion records, and aggregation and audit procedures supporting reproducibility (Reviewer AtcM).


Detailed responses are provided under each review. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Request for Reviewer Follow-up

Dear Area Chair,

We sincerely appreciate your time and efforts in handling our submission. We have taken the reviewers' concerns very seriously, conducted substantial additional experiments, carefully addressed each comment, and revised our manuscript accordingly. However, we have not yet received follow-up responses from the remaining reviewers. As the discussion deadline is approaching, we are concerned that there may be insufficient time for meaningful discussion of our new results. We would be very grateful if you could kindly encourage the remaining reviewers to consider our responses and provide feedback. We have devoted considerable effort to this work and sincerely hope that our revisions and additional evidence can receive careful consideration.

Thank you very much for your understanding and support.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.