- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a closed-loop post-training framework designed to tackle the multidimensional evaluation challenges inherent in video generation. By explicitly connecting capability-level feedback to targeted interventions across data collection, training, and inference, the authors propose a structured pathway toward recursive self-improvement (RSI).
Critical strengths
- The paper addresses an important problem: visual quality, reference fidelity, action adherence, and audio synchronization can move in different directions.
- The empirical results are quite interesting. The post-trained MiniMax-H3 model achieved an 85.3% advertising reference-fidelity pass rate, outperforming six frontier APIs. The Wan-2.2 post-trained system reduced absolute audio-visual offset from 6.25 to 1.69 frames.
Critical weaknesses
-
Benchmark coverage is too narrow. 17 advertising briefs whose prompts meet the strictest API character limit are too narrow.
-
Experiments and discussions do not show that the benchmarked prompts span the failures motivating the method, such as varied action sequences, long-range identity retention, speech timing, or audio–visual interactions.
-
The main evaluation is llm-judged, with no matching human study or human audits.
-
I find the novel contribution hard to identify. Video-T1 already uses verifier feedback to guide video-generation decisions, while prior post-training work combines data, rewards, prompting, and inference optimization. This paper’s possible distinction is using feedback to choose subsequent training interventions, but it neither specifies that policy clearly nor tests it against a comparable baseline.
-
Ablation studies or budget-comparable baselines are also missing.
-
While without clear evidence, many parts of the paper appear to be AI-generated, e.g., the Introduction. The reading flow is not very good.
Questions for authors
- At a fixed total training and evaluation budget, how does this policy compare with a fixed multi-stage recipe, random valid proposals, and an expert using the same diagnostic reports?
- Can the authors evaluate the final selected systems on fresh briefs using a blinded human panel, including the frontier API comparisons and the reference-fidelity threshold?
- How often did a change that improved its diagnosed capability pass the other capability guardrails and result in an actual promotion?
Training baselines, blinded human evaluation and candidate promotion
Q1. How does adaptive training compare with fixed, random and expert choices? Adaptive training led both fixed recipes on 24 held-out LTX prompts at the same training and evaluation budget. In paired video comparisons judged by Gemini 3.8 Flash, its preference score was 54.2% against the fixed multi-stage schedule and 52.1% against the constant recipe, with half credit for ties.
Each seed contributes 24 comparisons, so the two scores against random proposals average to the same value as the fixed schedule.
All methods started from the same LTX-2.3 weights and trained on the same 64 clips. We used two training seeds. Each run used 1,024 updates, 32 development videos and 64 development ratings. For each seed the judge saw all 24 test video pairs in both orders. Both 95% intervals against fixed recipes include 50%, so the lead on new prompts remains uncertain (Appendix R).
For the expert comparison, an author chose to lower the learning rate. We tested that choice for one 64-update round on each seed and retained the existing model both times. A full-budget expert comparison remains open (Appendix P).
Relation to Video-T1. Video-T1 uses verifier feedback to choose a generation path for the current video. We use feedback from one training run to decide what to change in the next. For H3, we found that the same video received inconsistent scores. We changed the training reward to use repeated rankings of videos for the same prompt. The post-trained model then scored 5.40 points above base on the original Gemini 3.5 Flash checklist across 20 briefs (Appendix F). We have revised the introduction to make this distinction clear and added an AI-use disclosure. Video-T1
Q2. What did the blinded comparisons show on fresh briefs? After selecting the post-trained H3 model and fixing the evaluator, we collected new human judgments. H3 won 5 of six comparisons against Seedance 2.5 (one tie) and 4 of six against H3 Max. Each API had its own six briefs. The full set of 24 new briefs includes objects disappearing and reappearing and speech timed to an action.
The paper’s 85.3% fidelity pass rate comes from a model evaluator giving at least 7/10 on 29 of 34 ratings. People judged overall preference in the blinded study. Human calibration of the 7/10 cutoff remains open. Video comparisons
How does H3 fare beyond advertising? On 238 general image-to-video prompts with audio, people chose post-trained H3 in 57.7% of 3,149 votes across 25 comparison models. In this blinded human review, 113 raters saw two videos made from the same reference and prompt with the model names hidden. Against Seedance 2.5, the vote was nearly even. People favored H3 over Veo 3.1 and Kling 3.0 Pro. Some votes from this larger study later informed evaluator design. Appendix Q reports all 25 comparisons with uncertainty by prompt.
Both post-trained H3 models also scored above 65% against their base in a separate blinded comparison on the same 29 briefs. Ties count half.
Q3. How often did an improvement lead to promotion? 1 of 22 candidates (4.5%) passed the other checks and replaced the incumbent after an initial gain on the LTX product-video test. The LTX study targeted product-video quality. We counted a gain when a candidate won more comparisons than it lost against the incumbent on its first test. Of those 22 candidates, 17 had complete checks for visual quality, prompt following and lip sync. 15 of 17 (88.2%) passed all three on every fully scored test, including retests. Each candidate counts once (Appendix I).