- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VideoTeach, a benchmark with 1,176 teaching cases across five domains. The benchmark evaluates instructional videos on three layers: video quality, teaching quality, and learning outcomes. Each case has an anchor problem plus a surface transfer problem and a structural transfer problem. The authors evaluate four video generation systems and report two findings. First, video quality and teaching quality scores are almost uncorrelated with learning gains (|r| <= 0.05). Second, all systems improve performance on the anchor problem (up to +8.3 pp), but transfer is weak: surface transfer peaks at +4.5 pp and structural transfer at +1.2 pp. This gap persists when using stronger teacher models (GPT-5.5, Gemini-3-Pro, Claude-Opus-4.7) and more teaching turns.
Critical strengths
- The three layer evaluation is a methodological advance. Previous benchmarks measure video quality, teaching quality, and learning separately, so they cannot test whether upstream quality actually leads to learning. The controlled transfer distances also let the authors separate memorizing the taught solution from understanding the underlying concept.
- The protocol includes several thoughtful controls: the NO-VIDEO baseline rules out retest effects, and the common-success subset separates generation reliability from output quality.
- The main finding is replicated across systems, teacher backbones, and interaction lengths. This consistency across independent experimental axes makes it unlikely to be an artifact of one particular setup.
Critical weaknesses
- My main concern is that all learning outcomes come from a single simulated student (Qwen2.5-VL-7B). The near-zero structural transfer gains may reflect the student model's limited ability rather than bad teaching. The 12 person human study is too small to settle this. I would suggest rerunning the main analysis with one or two additional student models of different capability (e.g., a 72B-class VLM), which should be feasible without changing the benchmark itself. Alternatively, the claims could be scoped more explicitly to simulated learners.
- The headline gains have no confidence intervals or significance tests. Some effects are only 1 to 4 pp, so it is hard to tell real effects from noise. Since case-level scores should already be logged, adding bootstrap confidence intervals or a paired test over cases would strengthen the paper at low cost. The correlation analysis would also be more convincing if computed per transfer problem instead of on averaged gains.
- GPT-5.5 serves as both evaluator and, in some experiments, teacher. This raises a self-preference concern for the absolute L1/L2 scores, although it does not affect the L3 gains and is therefore unlikely to change the main conclusions. A small human spot check of L1/L2 scores on a sample of cases would help calibrate the absolute values.
Questions for authors
- Were the main results obtained from a single run of the teacher, student, and evaluator models, or from multiple repetitions? If multiple, how much do the L3 gains vary across runs?
- How sensitive are the L1 scores to the two-second frame sampling used by the evaluator?
4 comments
Response to Reviewer kugW (Part 2/3)
W2: Uncertainty in learning gains and correlations by transfer target.
We added pointwise 95% paired-bootstrap confidence intervals for learning gains on the common-success subset, using 20,000 case-level resamples and preserving pre/post pairing.
| Method | (pp) | (pp) | (pp) |
|---|---|---|---|
| MONO | +8.32 [5.90, 10.53] | +2.84 [−1.14, 4.64] | −0.95 [−4.74, 2.85] |
| FREE | +5.48 [3.02, 8.13] | +0.95 [−3.04, 4.93] | +0.76 [−3.04, 4.55] |
| SCAF | +3.97 [1.51, 5.43] | +4.54 [0.76, 6.33] | −0.76 [−4.36, 2.84] |
| T2V | +0.76 [−0.95, 2.46] | −0.76 [−4.55, 3.03] | +1.13 pp [−1.14, 4.82] |
These results support our main finding: current code-based methods improve anchor performance, but transfer gains remain limited. All three code-based methods show anchor gains whose 95% confidence intervals exclude zero. Only SCAF shows a modest surface-transfer gain (+4.54 pp, 95% CI [0.76, 6.33]); no structural-transfer interval excludes zero.
We also compute within-system, item-level correlations separately for each target, without averaging gains:
| Method | Quality metric | with | with | with |
|---|---|---|---|---|
| MONO | L1: Video artifact | −0.007 | 0.042 | −0.033 |
| MONO | L2: Instructional process | 0.078 | −0.011 | 0.084 |
| FREE | L1: Video artifact | 0.009 | −0.020 | 0.066 |
| FREE | L2: Instructional process | −0.068 | −0.085 | −0.050 |
| SCAF | L1: Video artifact | −0.006 | −0.001 | 0.016 |
| SCAF | L2: Instructional process | 0.150 | -0.029 | -0.013 |
| T2V | L1: Video artifact | 0.006 | 0.034 | −0.078 |
| T2V | L2: Instructional process | −0.004 | −0.062 | 0.027 |
Across all the quality–learning comparisons, Pearson correlations remain close to zero (|r| ≤ 0.15). The weak quality–learning association persists when each learning target is analyzed separately.
We report the confidence intervals in Table 16 and target-specific correlations in Table 17 of Appendix H.
W3: Potential self-preference bias in GPT-5.5’s L1/L2 evaluations requires human calibration.
We randomly sample four cases from each of the five domains, yielding 20 cases and 60 videos generated by the three teacher backbones. Three human evaluators independently scored L1 and L2 using the same rubrics.
| Teacher backbone | L1: Evaluator | L1: Human | L2: Evaluator | L2: Human |
|---|---|---|---|---|
| GPT-5.5 | 67 | 81 | 75 | 70 |
| Claude Opus-4.7 | 79 | 90 | 65 | 63 |
| Gemini-3-Pro | 80 | 93 | 68 | 62 |
For L1, both evaluations support Gemini ≈ Claude > GPT-5.5. For L2, both support GPT-5.5 > Claude ≈ Gemini, with GPT-5.5 leading by 7–8 points under human assessment and 7–10 points under automatic evaluation. Thus, GPT-5.5’s discrepancy is not uniquely large, and this spot check provides no clear evidence that self-preference drives its leading L2 score.
Human L1 scores are consistently 11–14 points higher. This reflects different interpretations of rubric severity—for example, humans may assign smaller penalties to minor rendering defects. Importantly, the L1 ordering is unchanged. We have included the results in Appendix H.11.
Q1: Were the main results obtained from a single run of the teacher, student, and evaluator models, or from multiple repetitions? If multiple, how much do the L3 gains vary across runs?
The original results were obtained from a single run of the teacher–student–evaluator pipeline. We additionally performed three repetitions of the student pre-test and post-test evaluations on the same cases, keeping the generated teaching videos fixed. The table reports mean gains ± sample standard deviation across these three repetitions, in percentage points.
| Method | (pp) | (pp) | (pp) |
|---|---|---|---|
| MONO | +6.21 ± 2.93 | +2.82 ± 2.59 | +1.69 ± 1.69 |
| FREE | +5.04 ± 0.98 | +2.52 ± 3.18 | +1.13 ± 2.59 |
| SCAF | +3.39 ± 2.94 | +5.34 ± 2.59 | −2.82 ± 2.06 |
| T2V | +2.82 ± 2.18 | +1.52 ± 2.59 | −0.56 ± 0.98 |
Across methods, mean structural-transfer gains range from −2.82 to +1.69 pp, compared with positive anchor gains of +2.82 to +6.21 pp. Thus, the repeated evaluations retain the observed pattern of limited mean structural-transfer gains. We have included the results in Table 20 and Appendix H.12.
Response to Reviewer kugW (Part 3/3)
Q2: Sensitivity to frame sampling.
We rescore the same videos at 1-, 2-, and 4-second intervals, keeping the evaluator and rubric fixed. L1 scores are on a 0–100 scale.
| Sampling interval | MONO | FREE | SCAF | T2V | Mean L1 |
|---|---|---|---|---|---|
| 1 second | 49.34 | 55.20 | 46.13 | 40.54 | 47.80 |
| 2 seconds | 50.27 | 58.53 | 47.73 | 40.92 | 49.36 |
| 4 seconds | 53.34 | 60.27 | 51.47 | 42.26 | 51.83 |
Relative to 2-second sampling, mean L1 changes by −1.56 points at 1-second intervals and +2.47 points at 4-second intervals. The method ranking remains FREE > MONO > SCAF > T2V under all three settings. Thus, sampling affects absolute scores, but the observed system ranking is stable across the tested intervals. We have included this sensitivity analysis in the revised paper. We have included the results in Appendix H.13.
If the response addresses your concerns, could you consider increasing the score?
Response to Reviewer kugW (Follow up)
Dear Reviewer kugW,
Thank you for confirming that your concerns have been addressed. In light of this, would you consider raising your score?
Best regards,
The Authors
Response to Reviewer kugW (Part 1/3)
Dear Reviewer kugW,
Thank you for your feedback, we address your concerns as follows:
W1: Limited transfer may reflect the single student model’s capabilities rather than teaching limitations.
First, We add two student models, Gemma-3-4B-IT and LLaVA-OneVision-7B, alongside the original Qwen2.5-VL-7B-Instruct. All three were evaluated on the same 300 VideoTeach-Lite cases using MONO-generated teaching videos.
Gemma-3-4B-IT, Qwen2.5-VL-7B-Instruct, and LLaVA-OneVision-7B show anchor gains of +4.00 to +7.33 pp but no positive transfer gains despite substantial headroom. Thus, limited transfer is not unique to the original student.
Second, we test whether the original student can achieve transfer under improved teaching conditions.
We develop a human-guided video refinement framework. A Claude Opus-5.5 teacher explains the anchor solution and two worked examples using newly generated problem variants, separate from the benchmark’s transfer problems. Qwen2.5-VL-7B-Instruct watches the video and then answers these practice questions. A separate Claude Opus-5.5 critic reviews the video’s visual quality and the student’s answers, providing feedback to a Codex modifier using GPT-6.1-sol. The modifier revises the video-generation code over three rounds. A human then selects the best video from the initial version and the three revisions. Human feedback informed the framework’s design, while the final T1/T2 questions remained hidden throughout refinement and selection.
With Qwen2.5-VL-7B-Instruct fixed, results on the same 100 cases are:
Refinement increases surface-transfer gains by 7 pp and structural-transfer gains by 6 pp. The student therefore achieves greater transfer when the teaching condition is improved, indicating that the original videos’ limited gains cannot be explained solely by a fixed inability to learn from video.
Together, the multi-student evaluation and the fixed-student refinement comparison provide complementary evidence: limited transfer is neither unique to Qwen2.5-VL-7B-Instruct nor explained solely by its capabilities. We have included the experiments in Section 4.3 and Appendix H.9.