- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LabInstruct introduces 204 laboratory instructional video generation tasks across five disciplines, with real reference executions and procedure-specific checklists. Evaluations of eight generators using human and VLM judges reveal substantial gaps between visual consistency and procedural correctness.
Critical strengths
-
The benchmark targets a meaningful gap: whether generated videos correctly demonstrate laboratory procedures. Its action, contact, and state-transition annotations provide more useful diagnostics than generic visual-quality metrics.
-
Full human evaluation with four independent raters, two VLM judges, and multiple sensitivity analyses provides substantial support for the main finding that visual plausibility does not ensure procedural correctness.
Critical weaknesses
-
Automated evaluation remains insufficiently reliable for fine-grained comparisons: task-level human–VLM rank correlation is only 0.364, and LTX’s Overall score falls from 22.1 under VLM evaluation to 3.0 under human evaluation. Scoring real reference videos would help distinguish evaluator limitations from generation failures.
-
Completion becomes zero after two unmet Critical items, although one underlying failure may affect action, ordering, and state items simultaneously. The threshold sensitivity analysis does not establish robustness to checklist granularity or correlated failure counting.
-
Coverage is limited and imbalanced: 163 of 204 tasks come from Bilibili, and some operation categories contain only seven tasks. Without uncertainty estimates accounting for shared source videos, domain-specific strengths and small model differences are difficult to substantiate.
Questions for authors
Please refer to the critical weaknesses above.
3 comments
3. Source Imbalance and Statistical Uncertainty
Our conclusion is that LabInstruct's limited and uneven coverage restricts fine-grained claims, while the leading three-model group remains stable in full-benchmark sensitivity analyses. We explicitly acknowledge the coverage limitation in Appendix F (“Dataset Scale and Coverage”); Appendix B.1 and Table 8 document the selection criteria and recording provenance, and Appendix B.3 and Figure 6 provide the discipline and operation-domain breakdown.
Why is the dataset difficult to expand? A reference execution must show a continuous, sufficiently clear view of the same laboratory workspace, with task-relevant apparatus, object contacts, action order, and resulting state changes observable. Many publicly available laboratory videos contain cuts, camera changes, subtitles, overlays, or occlusions of critical actions; we also exclude unsafe demonstrations. Relaxing these criteria would increase the number of clips but undermine the visual verifiability needed for procedural evaluation. After screening, 81 source recordings yielded 204 tasks; some categories remain sparsely represented (e.g., 7 heating/cooling tasks). We do not claim that this curated set represents the full diversity of laboratories or procedures.
How should platform imbalance be interpreted? Table 8 shows that 61/81 recordings and 163/204 tasks were retrieved from Bilibili. These are provenance counts, not a measure of independent laboratories, creators, or experimental diversity: a hosting platform aggregates heterogeneous recordings, and the same platform does not imply the same scientific origin. Likewise, Bilibili-versus-other-platform score differences cannot isolate a platform effect because procedure, discipline, viewpoint, and recording style may differ. We therefore treat the non-Bilibili subset (41 tasks from 20 recordings) as an exploratory composition check, not evidence of platform-specific model performance.
Source-level robustness. We reweighted each source recording equally and performed leave-one-source-video-out (LOSO) analysis using the original scoring setup. The table compares the original task-weighted pooled GPT Overall scores (Table 7) with equal-source weighting:
| Model | Original GPT Overall | Equal-source-weighted GPT Overall |
|---|---|---|
| Wan 3.0 | 44.4 | 45.8 |
| Seedance 2.0 | 45.1 | 45.2 |
| MiniMax H3 | 46.3 | 44.8 |
The same three models remain the leading group in all 81 LOSO runs under both GPT and human evaluation, although their order changes under equal-source weighting. These analyses reduce the influence of multiple tasks from one recording; they do not eliminate broader collection bias or potential shared origins across recordings.
Domain-level uncertainty. We additionally used paired cluster bootstrapping at the source-recording level to assess within-domain GPT Overall differences:
| Domain | Tasks / Recordings | Comparison | Gap | 95% cluster-bootstrap CI |
|---|---|---|---|---|
| Heating/Cooling | 7 / 5 | Seedance 2.0 − MiniMax H3 | +20.2 | (−42.9, 83.3) |
| Instrument Operations | 60 / 28 | MiniMax H3 − Seedance 2.0 | +3.0 | (−9.5, 16.0) |
Neither interval excludes zero. Thus, the evidence supports stability of the leading group at the pooled-benchmark level, but not reliable within-tier or sparsely represented domain rankings. We will sharpen this distinction in the revision.
Thank you for the detailed rebuttal; the reference-video control, the operation-level grouping analysis, and the source-level bootstrap have addressed my concerns, so I will maintain my original score, and I encourage the authors to include these analyses in the revision and to qualify the domain-level rankings accordingly.
1. Fine-Grained Evaluation Reliability and the Reference-Video Control
We appreciate the reviewer’s suggestion. The task-level human–VLM rank correlation of 0.3 does point to limitations in automated evaluation for fine-grained comparisons, and we acknowledge this. The drop in LTX’s Overall score from 22.1 under VLM evaluation to 3.0 under human evaluation also reflects a broader pattern: the VLM tends to over-score weaker models.
Why are weaker models over-scored, particularly LTX 2.3?
Our analysis shows that the VLM can recognize the general scene and intended action, but often misses visual distortions, changes in object shape, and whether critical operations have actually been completed. These problems are especially pronounced in LTX 2.3.
First, the judge does not fully recognize the generation defects in weaker models. The equipment and broad outline of an operation may remain recognizable, even when tools deform, object boundaries merge, or contact and state changes are unclear. The VLM explanations nevertheless often describe the objects as intact and the operation as completed. Many actual failures in weaker models are consequently marked as successful. The Overall overestimation is 9.0–13.6 points for Wan 2.2, Cosmos3 Super, LingBot Video, and Cosmos3 Nano, compared with approximately 2.0 points for both MiniMax H3 and Seedance 2.0.
The largest discrepancy for LTX 2.3 concerns object consistency, which the VLM judge is notably insensitive to.
LTX has the largest object-consistency overestimation among all eight models. Of the 259 critical object-consistency checks that human raters marked as failed, the VLM marked 209 as YES. Its poor object persistence is therefore not adequately reflected in the automated score. These visual misses also lead to inflated judgments of execution. LTX’s average Completion score rises from 4.1 under human evaluation to 26.2 under VLM evaluation, while Overall rises from 3.0 to 22.1.
LTX is particularly over-scored because substantial structural distortions and object-shape drift go unnoticed. The judge then accepts the associated actions and completion states, leading to a systematic overestimation of procedural quality.
We have evaluated all 204 real reference videos. Their pooled Overall score is 93.1. We also conducted a final-step truncation control on 20 reference videos across five disciplines, with 10 videos at each task level. When each video was cut immediately before its final critical operation, the mean Overall score fell from 94.2 to 45.0, and Completion fell from 95.0 to 45.0. This shows that the judge lowers its score when at least some final steps are missing.
2. Completion and Correlated Failure Counting
We added an operation-level grouping analysis covering all 204 tasks and 8 models. We mapped the 1,971 Critical items so that items describing the same operation and its outcome are counted together. When a prerequisite action is not completed, failures in dependent items are not counted again. The table below reports pooled Overall scores before and after grouping.
GPT Overall increases by an average of 3.7 points across models, with gains ranging from 2.7 to 4.9 points. Human Overall increases by an average of 2.6 points, with gains ranging from 1.4 to 3.6 points. Correlated failure counting has a larger effect on partially completed Level 1 tasks: the mean increase in GPT Overall is 5.0 points for Level 1 and 2.4 points for Level 2.
The set of top-three models remains unchanged under both evaluation protocols. Seedance and MiniMax switch places in the GPT ranking, while the human ranking is unchanged. The Spearman rank correlations before and after grouping are 0.9 and 1.0, respectively. These results show that correlated item counting affects scores for partially completed tasks and can change the ordering of closely matched models. The leading group, however, remains stable after operation-level grouping.