- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Vibrics evaluates video generation pipelines using rubrics from the original request. It measures request fulfillment and video quality, and gives diagnostic reports. The benchmark contains 6,000 videos from five rewriters and five generators.
Critical strengths
- Using the original request is useful. It can find requirements that are lost during prompt rewriting.
- The crossed design helps compare rewriters and generators. The paper also includes human checks and separate calibration data.
Critical weaknesses
- The score can exclude requirements that were not checked. In C01, guzheng accompaniment is N/A, but Fulfillment is 1.000. It is unclear whether this requirement was verified elsewhere. Tables 17 and 18 also disagree: under the stated equal weighting, the Seedance–LTX Fulfillment gap is 0.1820 from Table 17, but 0.174 in Table 18.
- Most text-conditioned baselines receive the rewritten prompt, while Vibrics uses the original request. This can favor Vibrics. VideoArgus is an exception, but its rubric and judge settings are unclear. Also, the 89.6% diagnostic result only checks the main video problem. It does not show that Vibrics correctly assigns the problem to a pipeline stage.
Questions for authors
- Why is guzheng accompaniment N/A in C01 when Fulfillment is 1.000? How do unchecked requirements affect the score?
- Why do Tables 17 and 18 give different contrasts? Do they use the same data and weights?
- How do the results change if baselines receive the same original request and references?
- Was stage-level diagnosis checked by human annotators, beyond finding the main video problem?
4 comments
Dear Reviewers,
We would like to kindly and respectfully follow up, as the rebuttal period is drawing to a close. In addition to our response to the reviews, we have also submitted a revised version of the paper along with a summary of the changes. We have carefully addressed all raised concerns, and we would greatly appreciate your feedback whenever you have the opportunity.
Thank you very much for your time and consideration.
Best regards,
The Authors
Request for Reviewer Feedback Before the Discussion Deadline
Dear Reviewer a19T,
As the discussion deadline is approaching, we understand that you may be very busy. If you could spare some time to share your thoughts, we would sincerely appreciate it and would be very happy to discuss any remaining questions in greater detail.
We are encouraged that Reviewer AtcM has kindly followed up and reaffirmed their positive assessment of our paper. We would also greatly value your feedback on whether our responses, additional experiments, and revisions have adequately addressed your concerns.
Thank you very much for your time and consideration.
Best regards, The Authors
Response
Some of my concerns have been addressed, so I am willing to keep my original score. I still have reservations regarding the level of innovation, but if the paper is accepted, I have no objection.
Response to Reviewer a19T
Thank you for closely checking the C01 example, the aggregation results, and the baseline evaluation protocol. We corrected the affected audio judgment and ranking tables, reran input-controlled comparisons, and added expert checks of the judgments used to diagnose failures at different stages.
W1 & Q1 & Q2
The C01 example marks explicitly requested guzheng audio as N/A despite a perfect Fulfillment score, and the Seedance–LTX differences are inconsistent across two tables. What caused these errors, and how were they corrected?
Response
We rechecked C01 and found that the Omni audio-evaluator call had failed because of an input-formatting error. The failure to obtain a judgment did not make the requested audio inapplicable. Inspection of the delivered video confirmed that it contains no audio track, so the explicit guzheng-accompaniment requirement must be marked FAIL rather than N/A. Including this applicable criterion changes Fulfillment from 1.00 to 37/38 = 0.97; Craft remains 0.68, and uncalibrated Utility becomes 0.86. We also reviewed historical audio-evaluator failures and propagated verified corrections through the affected scores and rankings.
For the inconsistent Seedance–LTX comparison, the original tables used different effective sets of valid records and different aggregation orders: one summarized pipeline-combination means, while the other averaged within-request contrasts. We recalculated both from the same valid request-level records, giving each valid request equal weight within a combination and each rewriting method equal weight when comparing generators. The contrast now comes from the unrounded combination means, producing a Seedance–LTX Fulfillment difference of 0.18 in both summaries after rounding. The leading rewriter and generator remain unchanged after the corrections.
The revised results now apply one consistent rule for criterion applicability and one consistent aggregation protocol across the case analysis and benchmark comparisons. Appendices A.2 and F.1 (Table 28) document the corrected C01 case; Appendix D (Tables 22–23) reports the corrected matrix and request-clustered intervals.
W2 & Q3 & Q4
Are external evaluators compared with suitable access to the original request and references? Does the 89.6% diagnostic result also validate judgments about where a requirement was lost?
Response
We reran five text-conditioned evaluators on the same 40 requests and 1,000 human-rated videos, giving each either the original request or the rewritten generation prompt. The table reports request-averaged Kendall's τ-b against expert overall ratings:
Four of the five text-conditioned baselines improve with the original request. VideoArgus constructs its own request-specific rubric, while Vibrics derives its rubric from the original request and any references provided. We report the actual reference access of each method. More directly, our matched agent comparison supplies both free-form and Vibrics-guided agents with the same original request, references, candidate video, backbone, and tools. With Flash, overall-preference ordering accuracy improves from 61.62% to 71.77% when the agent uses Vibrics's shared criteria and evidence. This shows the benefit of the evaluation procedure beyond simply changing the source text.
We also clarified the two forms of diagnostic validation. The reported 89.6% measures identification of the most important problem in the final video. In a separate two-expert audit, the criterion judgments supporting stage-level diagnosis agree with expert consensus at 87.23% for rewritten prompts, 80.00% for available process traces, and 84.62% for delivered videos. These results validate both the evaluator's ability to identify the main problem and the stage-specific evidence used to explain it.
The corrected input comparison and agent results appear in Section 4.1 and Table 3; Section 4.2 and Appendices C.1 and C.5 document the distinct diagnostic checks and their agreement with experts.