acceptodds
Under review as a conference paper at ICLR 2027

LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

Abstract

Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.

Primary area:
datasets and benchmarks
Keywords:
Self-driving Lab, Instruction Video, Video Generation, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_oMqrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LabInstruct introduces 204 laboratory instructional video generation tasks across five disciplines, with real reference executions and procedure-specific checklists. Evaluations of eight generators using human and VLM judges reveal substantial gaps between visual consistency and procedural correctness.

Critical strengths
  • The benchmark targets a meaningful gap: whether generated videos correctly demonstrate laboratory procedures. Its action, contact, and state-transition annotations provide more useful diagnostics than generic visual-quality metrics.

  • Full human evaluation with four independent raters, two VLM judges, and multiple sensitivity analyses provides substantial support for the main finding that visual plausibility does not ensure procedural correctness.

Critical weaknesses
  • Automated evaluation remains insufficiently reliable for fine-grained comparisons: task-level human–VLM rank correlation is only 0.364, and LTX’s Overall score falls from 22.1 under VLM evaluation to 3.0 under human evaluation. Scoring real reference videos would help distinguish evaluator limitations from generation failures.

  • Completion becomes zero after two unmet Critical items, although one underlying failure may affect action, ordering, and state items simultaneously. The threshold sensitivity analysis does not establish robustness to checklist granularity or correlated failure counting.

  • Coverage is limited and imbalanced: 163 of 204 tasks come from Bilibili, and some operation categories contain only seven tasks. Without uncertainty estimates accounting for shared source videos, domain-specific strengths and small model differences are difficult to substantiate.

Questions for authors

Please refer to the critical weaknesses above.

3 comments

1. Fine-Grained Evaluation Reliability and the Reference-Video Control

We appreciate the reviewer’s suggestion. The task-level human–VLM rank correlation of 0.3 does point to limitations in automated evaluation for fine-grained comparisons, and we acknowledge this. The drop in LTX’s Overall score from 22.1 under VLM evaluation to 3.0 under human evaluation also reflects a broader pattern: the VLM tends to over-score weaker models.

Why are weaker models over-scored, particularly LTX 2.3?

Our analysis shows that the VLM can recognize the general scene and intended action, but often misses visual distortions, changes in object shape, and whether critical operations have actually been completed. These problems are especially pronounced in LTX 2.3.

First, the judge does not fully recognize the generation defects in weaker models. The equipment and broad outline of an operation may remain recognizable, even when tools deform, object boundaries merge, or contact and state changes are unclear. The VLM explanations nevertheless often describe the objects as intact and the operation as completed. Many actual failures in weaker models are consequently marked as successful. The Overall overestimation is 9.0–13.6 points for Wan 2.2, Cosmos3 Super, LingBot Video, and Cosmos3 Nano, compared with approximately 2.0 points for both MiniMax H3 and Seedance 2.0.

The largest discrepancy for LTX 2.3 concerns object consistency, which the VLM judge is notably insensitive to.

ModelHuman object consistencyVLM object consistencyOverestimation
LTX 2.348.785.236.5
LingBot Video67.690.723.0
Cosmos3 Nano73.692.618.9

LTX has the largest object-consistency overestimation among all eight models. Of the 259 critical object-consistency checks that human raters marked as failed, the VLM marked 209 as YES. Its poor object persistence is therefore not adequately reflected in the automated score. These visual misses also lead to inflated judgments of execution. LTX’s average Completion score rises from 4.1 under human evaluation to 26.2 under VLM evaluation, while Overall rises from 3.0 to 22.1.

LTX is particularly over-scored because substantial structural distortions and object-shape drift go unnoticed. The judge then accepts the associated actions and completion states, leading to a systematic overestimation of procedural quality.

We have evaluated all 204 real reference videos. Their pooled Overall score is 93.1. We also conducted a final-step truncation control on 20 reference videos across five disciplines, with 10 videos at each task level. When each video was cut immediately before its final critical operation, the mean Overall score fell from 94.2 to 45.0, and Completion fell from 95.0 to 45.0. This shows that the judge lowers its score when at least some final steps are missing.

2. Completion and Correlated Failure Counting

We added an operation-level grouping analysis covering all 204 tasks and 8 models. We mapped the 1,971 Critical items so that items describing the same operation and its outcome are counted together. When a prerequisite action is not completed, failures in dependent items are not counted again. The table below reports pooled Overall scores before and after grouping.

ModelGPT, originalGPT, groupedHuman, originalHuman, grouped
MiniMax H346.349.144.347.9
Seedance 2.045.149.143.146.6
Wan 3.044.548.137.339.7
Wan 2.225.929.715.418.6
Cosmos3 Super22.826.511.514.5
LTX 2.322.125.13.04.5
LingBot Video19.624.510.512.0
Cosmos3 Nano19.423.55.88.4

GPT Overall increases by an average of 3.7 points across models, with gains ranging from 2.7 to 4.9 points. Human Overall increases by an average of 2.6 points, with gains ranging from 1.4 to 3.6 points. Correlated failure counting has a larger effect on partially completed Level 1 tasks: the mean increase in GPT Overall is 5.0 points for Level 1 and 2.4 points for Level 2.

The set of top-three models remains unchanged under both evaluation protocols. Seedance and MiniMax switch places in the GPT ranking, while the human ranking is unchanged. The Spearman rank correlations before and after grouping are 0.9 and 1.0, respectively. These results show that correlated item counting affects scores for partially completed tasks and can change the ordering of closely matched models. The leading group, however, remains stable after operation-level grouping.

3. Source Imbalance and Statistical Uncertainty

Our conclusion is that LabInstruct's limited and uneven coverage restricts fine-grained claims, while the leading three-model group remains stable in full-benchmark sensitivity analyses. We explicitly acknowledge the coverage limitation in Appendix F (“Dataset Scale and Coverage”); Appendix B.1 and Table 8 document the selection criteria and recording provenance, and Appendix B.3 and Figure 6 provide the discipline and operation-domain breakdown.

Why is the dataset difficult to expand? A reference execution must show a continuous, sufficiently clear view of the same laboratory workspace, with task-relevant apparatus, object contacts, action order, and resulting state changes observable. Many publicly available laboratory videos contain cuts, camera changes, subtitles, overlays, or occlusions of critical actions; we also exclude unsafe demonstrations. Relaxing these criteria would increase the number of clips but undermine the visual verifiability needed for procedural evaluation. After screening, 81 source recordings yielded 204 tasks; some categories remain sparsely represented (e.g., 7 heating/cooling tasks). We do not claim that this curated set represents the full diversity of laboratories or procedures.

How should platform imbalance be interpreted? Table 8 shows that 61/81 recordings and 163/204 tasks were retrieved from Bilibili. These are provenance counts, not a measure of independent laboratories, creators, or experimental diversity: a hosting platform aggregates heterogeneous recordings, and the same platform does not imply the same scientific origin. Likewise, Bilibili-versus-other-platform score differences cannot isolate a platform effect because procedure, discipline, viewpoint, and recording style may differ. We therefore treat the non-Bilibili subset (41 tasks from 20 recordings) as an exploratory composition check, not evidence of platform-specific model performance.

Source-level robustness. We reweighted each source recording equally and performed leave-one-source-video-out (LOSO) analysis using the original scoring setup. The table compares the original task-weighted pooled GPT Overall scores (Table 7) with equal-source weighting:

ModelOriginal GPT OverallEqual-source-weighted GPT Overall
Wan 3.044.445.8
Seedance 2.045.145.2
MiniMax H346.344.8

The same three models remain the leading group in all 81 LOSO runs under both GPT and human evaluation, although their order changes under equal-source weighting. These analyses reduce the influence of multiple tasks from one recording; they do not eliminate broader collection bias or potential shared origins across recordings.

Domain-level uncertainty. We additionally used paired cluster bootstrapping at the source-recording level to assess within-domain GPT Overall differences:

DomainTasks / RecordingsComparisonGap95% cluster-bootstrap CI
Heating/Cooling7 / 5Seedance 2.0 − MiniMax H3+20.2(−42.9, 83.3)
Instrument Operations60 / 28MiniMax H3 − Seedance 2.0+3.0(−9.5, 16.0)

Neither interval excludes zero. Thus, the evidence supports stability of the leading group at the pooled-benchmark level, but not reliable within-tier or sparsely represented domain rankings. We will sharpen this distinction in the revision.

Thank you for the detailed rebuttal; the reference-video control, the operation-level grouping analysis, and the source-level bootstrap have addressed my concerns, so I will maintain my original score, and I encourage the authors to include these analyses in the revision and to qualify the domain-level rankings accordingly.

Reviewer_t3MzOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces LabInstruct, a benchmark for evaluating the procedural correctness of instructional video generation in real-world laboratory settings. It consists of 204 tasks across five scientific disciplines, with reference videos and structured annotations of actions, objects, and state changes. The benchmark provides a procedure-aware evaluation protocol covering visual quality, task completion, and validity constraints. Experiments on eight video generation models, supported by both automated and human evaluations, reveal a substantial gap between visual realism and procedural correctness, highlighting key challenges for reliable laboratory instruction generation.

Critical strengths

A. Novel and Timely Research Direction. The paper explores instructional video generation in real-world laboratory settings, offering a novel perspective on the increasingly important topic of self-driving laboratories. Investigating whether AI-generated instructional videos can support humans in conducting scientific experiments is a meaningful and underexplored research direction, with potential implications for future embodied laboratory automation.

B. Comprehensive Benchmark and Experimental Analysis. The paper presents a well-designed benchmark covering diverse laboratory tasks and scientific disciplines, along with comprehensive evaluations of existing video generation models. The detailed experimental results and analyses reveal important limitations of current models, particularly the gap between visual realism and procedural correctness, providing valuable insights and directions for future research.

Critical weaknesses

I did not identify any major weaknesses that would undermine the paper's contributions. However, I have a few minor concerns and questions that I would appreciate the authors addressing in the rebuttal or discussing in the revised paper. Please refer to the Questions for Authors section below.

Questions for authors

A. The connection between instructional video generation and self-driving laboratories is not sufficiently clear. Could the authors further clarify this connection to better justify the motivation and significance of the benchmark?

B. Based on the experimental findings, what are the most critical limitations that future video generation models should prioritize addressing? A discussion of these directions would strengthen the paper's impact.

2 comments

1. Clarification of SDL Relevance

The link to SDLs is the gap between experiment planning and reliable physical execution. Even capable embodied systems face difficulties with precise, contact-rich manipulation, physical-state verification, failure recovery, and maintaining progress over long procedures. For example, recent GPT-6-Astra studies identify these challenges alongside practical execution-time and deployment costs (https://arxiv.org/abs/2610.22471). Heterogeneous laboratory apparatus and safety constraints make fully autonomous execution especially challenging, so researchers remain involved in carrying out some experimental steps.

Consider a hypothetical SDL plan to transfer liquid from tube A to tube B. A workspace-grounded video would need to show the correct source and destination, aspiration before dispensing, and the intended observable final state—not merely a plausible-looking pipetting motion. Given a real initial workspace image and an instruction, LabInstruct tests whether a generated demonstration satisfies these procedural requirements. Its task specifications and instructions are derived from human-verified recordings, not an AI planner; the reference executions provide verifiable ground truth.

Thus, LabInstruct benchmarks the reliability of a potential human-facing instructional step in an SDL, rather than evaluating an end-to-end SDL, robotic control, or the effectiveness of videos in actual human experiments. Generated demonstrations would require independent expert and SOP review before any real-world use. We will clarify this distinction in the abstract and introduction.

2. Priorities for Future Video Generation Models

Our results point to two priorities: procedural fidelity across dependent actions, and persistent, verifiable object states throughout a generation.

First, improve action–state fidelity and multi-step completion. Models often preserve a plausible scene while omitting, misordering, or misdirecting critical operations and failing to realize the required state changes (Sec. 4.3). Under human evaluation, even the highest Level 2 Completion score is only 43.6/100 (Table 5). Future models should explicitly track target objects, contacts, temporal dependencies, and the observable outcome of each step.

Second, improve interaction-time consistency and single-generation reliability. Deformed tools, merged object boundaries, and unstable object identities can make critical actions impossible to verify. Generation randomness is another limitation: in our 40-task best-of-three analysis, Seedance 2.0 and Wan 3.0 gain 14.0 and 11.1 Overall points, respectively (Table 20). Future methods should reduce these structural failures and reliably produce complete demonstrations without relying on multiple attempts.

The goal is not simply more realistic video, but demonstrations whose actions, ordering, contacts, and final states can be checked against the requested procedure. We will make these priorities more explicit in the conclusion.

Thank you for your response; I have no further questions. Please include these clarifications in the camera-ready version, and consider releasing your code and data as open-source soon.

Reviewer_9PiHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LabInstruct, a benchmark to evaluate whether an image-to-video model can generate a correct lab procedure video. The input is the first frame of a real lab workspace and one text instruction. The model needs to generate a video that really does the procedure, not only looks realistic.

The benchmark has 204 tasks from 81 source videos (from Bilibili, YouTube, FineBio, ExpVid), 5 disciplines, and two levels. Level 1 is a single-step task (4s); Level 2 is a short multi-step task (12s). Each task includes a real reference video, a structured task spec, and a human-verified checklist.

Scoring uses importance-weighted violations with inverse decay, geometric mean over 4 core dimensions, a critical completion factor, and two binary gates (scene, safety). The authors test 8 I2V models with GPT-5.6 Sol as judge, and also do full human evaluation with 4 raters. The main finding is that models keep objects and scenes quite well, but action and state changes are often wrong. The best pooled overall performance is only about 46. The paper also includes many robustness analyses.

Critical strengths
  • The problem is clear. Visual plausibility does not totally reflect procedural correctness, and this point is well illustrated in Figure 1.
  • It uses real lab video with reference execution from the same workspace. This is distinct from previous science video benchmarks.
  • The data collection and annotation pipeline seems to be careful and comprehensive.
  • The checklist design is reasonable. Include atomic, positively phrased, visually verifiable, with Critical/Standard/Supplementary weight. And lots of effort on human evaluation of all collected videos and proposed items.
Critical weaknesses

Motivation

The paper's motivation paragraph is a little puzzling to me. The SDL story is not connected well to what the benchmark actually does. The abstract and Intro say AI-generated plan → generated video → human execute. But in the benchmark, tasks come from human-recorded routine operations (zero a balance, move hanger, pipetting). Instructions are not from any AI planner; they are rendered from reference video spec. Also, the abstract spends too much on motivation and leaves only 2-3 sentences for the actual benchmark and findings.

Evaluation

  • Human-VLM agreement may look better than real. Pearson 0.980 is on 8 points, and 8 models clearly separate into two clusters (3 strong, 5 weak); this makes a high Pearson score easy. Level 2 Spearman is only 0.523; task-level ρ is only 0.364. VLM is also too lenient for weak models (LTX 2.3: 22.1 vs human 3.0).
  • No positive control. What is the score if you give the reference video itself to judge? This can tell the ceiling of the checklist + judge.
  • MiniMax H3 L1 videos are 5.2s; the judge gets 21 frames, other models get 16 frames. Your FPS study shows GPT gives a higher score with more frames. So MiniMax L1 lead is maybe confounded.
Questions for authors
  1. Could you clarify with one concrete SDL usage scenario that makes the proposed benchmark necessary?
  2. What is the smallest score gap the VLM judge can reliably separate? Can the judge be used to compare models inside the same tier? Why is the VLM much more lenient for weak models (e.g., LTX 2.3: 22.1 vs. 3.0)?
  3. What score does the reference video itself get under the same checklist and judge? A negative control, such as a reference clip truncated before the last critical step, would also help show whether the judge catches missing actions.
  4. MiniMax H3 L1 videos are 5.2 s and get 21 judged frames, while other models get 16. If you evaluate only the first 4 s of MiniMax H3 videos, or fix the number of frames for all models, does its Level 1 lead remain?
3 comments

1. Clarification on SDL Motivation and Relevance

Thank you for highlighting the gap in our SDL motivation. Despite recent advances, reliable physical execution remains a bottleneck in SDLs. For example, studies of GPT-6-Astra report persistent challenges in precise manipulation, physical-state verification, long-horizon progress tracking, and recovery from failures (https://arxiv.org/abs/2610.22471). Diverse apparatus, contact-rich operations, and safety requirements compound these challenges in real laboratories, where researchers still perform steps that cannot yet be reliably delegated to robots. This motivates workspace-grounded instructional videos for human-in-the-loop execution. LabInstruct evaluates the procedural correctness of those videos, not experiment planning or robot control.

Consider a hypothetical SDL liquid-transfer step:

[SDL plan: transfer A -> B]
              |
              v
[Workspace image + instruction]
              |
              v
       [I2V demonstration] ---> [Human execution]
              |
              v
    [LabInstruct (offline)]
  correct A -> B | action order | final state

Here, the SDL workflow is an intended use case, not our experimental pipeline. LabInstruct's instructions are derived from human-verified specifications of real recorded procedures, not generated by an AI planner; the paired references and checklists enable controlled evaluation of video-level procedural fidelity. Human execution would require independent expert/SOP review. We will clarify this boundary in the abstract and introduction and shorten the abstract's background motivation.

2. Human–VLM Consistency and Its Scope of Applicability

The reviewer is right that a high overall correlation does not, by itself, show that the VLM judge can distinguish between models with similar performance. We therefore added pairwise score-gap analyses for all models and examined the item-level judgments for LTX 2.3.

We ran 50,000 paired cluster-bootstrap resamples of the existing scores. The 204 tasks come from 81 source videos. Since tasks from the same video share the workspace, equipment, and recording conditions, we resampled at the source-video level and included all tasks from each selected video. The same resamples were used for every model when calculating score differences. We report differences in GPT-judge pooled Overall scores, with Bonferroni-adjusted intervals across all 28 comparisons. The family-wise nominal error rate is controlled at 5%.

2.1 Can the judge compare models within the same tier?

The 7 within-tier comparisons are listed below.

TierModel comparisonGapAdjusted interval
T0MiniMax H3 − Seedance 2.01.2[−11.8, 13.5]
T0MiniMax H3 − Wan 3.01.9[−9.9, 13.4]
T0Seedance 2.0 − Wan 3.00.6[−11.1, 12.8]
T1Wan 2.2 − Cosmos3 Super3.1[−8.7, 15.2]
T2LTX 2.3 − LingBot Video2.5[−7.5, 12.2]
T2LTX 2.3 − Cosmos3 Nano2.7[−8.4, 12.9]
T2LingBot Video − Cosmos3 Nano0.1[−8.9, 8.9]

All seven intervals include zero. Their unadjusted 95% intervals also include zero. The current results therefore do not support a stable ranking among models within the same tier.

2.2 Are the differences across tiers reliable? What is the smallest reliable gap?

The 21 cross-tier comparisons are shown below.

Tier pairModel comparisonGapAdjusted interval
T0–T1MiniMax H3 − Wan 2.220.4[8.2, 31.6]
T0–T1MiniMax H3 − Cosmos3 Super23.5[12.7, 33.6]
T0–T2MiniMax H3 − LTX 2.324.2[13.2, 35.4]
T0–T2MiniMax H3 − LingBot Video26.7[12.9, 39.6]
T0–T2MiniMax H3 − Cosmos3 Nano26.9[13.2, 39.8]
T0–T1Seedance 2.0 − Wan 2.219.1[6.5, 31.9]
T0–T1Seedance 2.0 − Cosmos3 Super22.2[11.0, 33.9]
T0–T2Seedance 2.0 − LTX 2.322.9[10.9, 34.8]
T0–T2Seedance 2.0 − LingBot Video25.5[13.4, 37.5]
T0–T2Seedance 2.0 − Cosmos3 Nano25.6[13.2, 38.5]
T0–T1Wan 3.0 − Wan 2.218.5[7.0, 30.3]
T0–T1Wan 3.0 − Cosmos3 Super21.6[9.3, 34.5]
T0–T2Wan 3.0 − LTX 2.322.3[10.4, 35.0]
T0–T2Wan 3.0 − LingBot Video24.9[12.1, 37.9]
T0–T2Wan 3.0 − Cosmos3 Nano25.0[10.1, 39.9]
T1–T2Wan 2.2 − LTX 2.33.8[−6.6, 14.1]
T1–T2Wan 2.2 − LingBot Video6.3[−2.4, 15.1]
T1–T2Wan 2.2 − Cosmos3 Nano6.5[−4.5, 17.2]
T1–T2Cosmos3 Super − LTX 2.30.6[−9.7, 11.3]
T1–T2Cosmos3 Super − LingBot Video3.2[−8.3, 14.3]
T1–T2Cosmos3 Super − Cosmos3 Nano3.4[−8.2, 14.6]

The 15 comparisons between T0 and the other models show gaps of 18.5–26.9 points, all of which remain statistically significant after correction for multiple comparisons. The analysis also provides some evidence for differences between closer models. For example, Wan 2.2 scores 6.3 points higher than LingBot Video. These results suggest that the judge can distinguish the leading tier from the remaining models and can support some comparisons between models with closer scores.

We cannot define a universal minimum reliable gap from these results. The 18.5-point gap is simply the smallest observed gap among the significant comparisons, not a resolution threshold for the judge. Uncertainty depends on the model pair and task composition, and may also reflect stochastic variation in the VLM judge.

2.3 Why are weak models over-scored, particularly LTX 2.3?

Our item-level results and inspection of generated videos suggest that weak models are over-scored because the VLM can recognize the general scene and intended action but often misses visual distortions, changes in object shape, and whether critical operations have actually been completed. These problems are especially pronounced in LTX 2.3.

First, the judge does not fully recognize the generation defects in weaker models. The equipment and broad outline of an operation may remain recognizable, even when tools deform, object boundaries merge, or contact and state changes are unclear. The VLM explanations nevertheless often describe the objects as intact and the operation as completed. Many actual failures in weaker models are consequently marked as successful. The Overall overestimation is 9.0–13.6 points for Wan 2.2, Cosmos3 Super, LingBot Video, and Cosmos3 Nano, compared with approximately 2.0 points for both MiniMax H3 and Seedance 2.0.

The largest discrepancy for LTX 2.3 concerns object consistency, which the VLM judge is notably insensitive to.

ModelHuman object consistencyVLM object consistencyOverestimation
LTX 2.348.785.236.5
LingBot Video67.690.723.0
Cosmos3 Nano73.692.618.9

LTX has the largest object-consistency overestimation among all eight models. Of the 259 critical object-consistency checks that human raters marked as failed, the VLM marked 209 as YES. Its poor object persistence is therefore not adequately reflected in the automated score.

The same visual misses also lead to inflated judgments of execution. For LTX, the VLM marked 663 of the 1,225 critical core items that humans labeled NO or UNJUDGEABLE as YES. Among the 190 videos with a human Completion score of zero, the VLM assigned a nonzero Completion score to 59. Consequently, the average Completion score for LTX rises from 4.1 under human evaluation to 26.2 under VLM evaluation, while Overall rises from 3.0 to 22.1.

LTX is particularly over-scored because substantial structural distortions and object-shape drift go unnoticed. The judge then accepts the associated actions and completion states, leading to a systematic overestimation of procedural quality.

3. Reference Videos as a Positive Control

We added an evaluation of the real reference videos. The results are shown below.

MetricLevel 1Level 2
Core Score98.398.8
Completion94.596.1
Scene gate100.0100.0
Safety gate98.098.0
Overall92.194.0

The pooled Overall score across all tasks is 93.0. The reference videos achieve near-ceiling Core and Overall scores at both levels and score well above all generated models. They provide a positive control based on real executions.

We also conducted a truncation experiment on 20 reference videos, covering five disciplines with 10 videos at each task level. When each video was cut immediately before its final critical operation, the mean Overall score fell from 94.2 to 45.0, and Completion fell from 95.0 to 45.0. This indicates that the judge lowers its score when at least some final steps are missing.

4. Temporal Coverage Bias in MiniMax H3

We followed the reviewer’s suggestion and uniformly sampled 16 frames from each complete MiniMax H3 video across all 101 Level 1 tasks. We kept the checklist, judging prompt, GPT-5.6 Sol, and scoring formula unchanged, and compared the results with the existing 16-frame evaluations of the other models.

With the frame count fixed, MiniMax H3’s Overall score drops from 53.6 to 30.1, and its rank falls from first to fourth. It is now below Wan 3.0 (46.2), Seedance 2.0 (41.9), and Wan 2.2 (38.5). Completion drops from 58.4 to 34.6, a larger decrease than Core, which falls from 82.6 to 73.1. This suggests that the assessment of critical-step completion is particularly sensitive to the sampling strategy.

MiniMax H3’s shortest native video is 5.2 seconds (124 frames). Sampling 16 frames from it gives a density of about 3.1 frames per second, while the 4-second videos from other models are sampled at 4 frames per second. The sparser sampling may miss brief critical actions. Simply truncating MiniMax H3 videos to the first four seconds could also remove steps that occur later. MiniMax H3 therefore loses its Level 1 lead under fixed 16-frame evaluation, but this result does not establish that it performs worse than the other models when generation duration is held constant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.