- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces V-ICAL, a benchmark that tests whether multimodal agents can learn task rules and strategies from video demonstrations and use them in closed-loop interaction. It contains 342 tasks built from 37 Gym environments. For each task, the agent receives 1–4 captioned demonstration clips, minimal rule text and the action space, and then interacts step by step from pixels. The authors evaluate 19 models along with random and human baselines; the best model, Seed-2.1-Pro, scores 54.4, against 83.6 for humans. An LLM auditor attributes failures with a four-stage model, with 95.0% agreement in human verification. Finally, a video/rule ablation shows that the gains from video are inconsistent.
Critical strengths
- The direction is a valuable one. The paper brings video in-context learning from offline QA into closed-loop interactive evaluation, filling a gap in how this ability is evaluated. The finding that current models cannot yet reliably benefit from video demonstrations is itself useful, as it points to a clear open problem.
- The benchmark is carefully built and reproducible. Environments are screened with explicit criteria, start states come from human play and reach decision-heavy situations rarely seen at reset, scoring is calibrated per task, and the authors commit to releasing raw trajectories. As video understanding moves from offline QA toward interactive agents, a closed-loop testbed like this is of real value to the community.
- The failure analysis goes beyond a single score and is verified by blind human checks. The finding that failures come mostly from state understanding and planning rather than action execution is useful for model developers.
Critical weaknesses
W1. The paper has not shown that the benchmark measures video in-context learning. The authors' own ablation (§4.4, Table 4) shows that removing the videos changes scores only slightly and not in a consistent direction; for example, under base rules Gemini-3-Flash scores higher without video (41.2 vs 39.6), and with detailed rules the video/no-video gap is within 1.6 points for all four models. The Conclusion also states that demonstrations "yield no consistent aggregate gains". In addition, the ablation does not include the top-ranked models, and the prompt gives the game name directly (App. B.3). Many of these are well-known games such as Pong and Tetris, so the models' prior knowledge may play a large role. Given this, the leaderboard in Table 1 seems to measure game-playing from priors and rules more than the ability suggested by the title and abstract.
W2. The main results lack uncertainty estimates. Each model is run once per task, and neither Table 1 nor Table 4 reports confidence intervals, while many ranking gaps and ablation differences are only 1–2 points and may fall within noise. The two main memory/dynamics conclusions face the same issue: Single-memory has only 4 environments and Static only 10, and they largely correspond to particular environment families, so it is currently hard to separate these conclusions from the difficulty of the environments themselves.
None of these issues undermines the value of the benchmark as a resource. If the authors can add no-video/name-masked controls for the strong models with confidence intervals, I would be more confident in supporting the paper.
Questions for authors
- Could you report results for the top-ranked models without video and with the game name masked, and use the "video gain" (with video minus without video) as a primary metric? A consistent, significant gain for the strong models would make the benchmark's framing much more convincing; if there is none, the paper might be reframed around the finding that current models cannot yet make effective use of video demonstrations.
- Could you add environment-clustered bootstrap confidence intervals to Tables 1 and 4, and report each model's decoding settings (temperature, reasoning budget, etc.)?
- Could the memory/dynamics conclusions be tested within the same environment family to rule out environment difficulty?
- The prompt tells the model that the demonstrations are "not necessarily good" play and are only for understanding environment dynamics (App. B.3). Could this affect the strategy-transfer results in Table 2? Did you try a prompt that encourages imitating strategies?
7 comments
1. No-Video Controlled Ablation & Video Gain Analysis for Strong Models (Response to W1, Q1 Part 1)
W1: The paper has not shown that the benchmark measures video in-context learning. The authors' own ablation shows that removing the videos changes scores only slightly and not in a consistent direction; for example, under base rules Gemini-3-Flash scores higher without video , and with detailed rules the video/no-video gap is within 1.6 points for all four models. The Conclusion also states that demonstrations "yield no consistent aggregate gains". In addition, the ablation does not include the top-ranked models, ...
Q1: Could you report results for the top-ranked models without video and with the game name masked, and use the "video gain" (with video minus without video) as a primary metric? A consistent, significant gain for the strong models would make the benchmark's framing much more convincing; if there is none, the paper might be reframed around the finding that current models cannot yet make effective use of video demonstrations.
We appreciate your insightful observation in W1 and Q1 that our initial video/no-video ablation (Table 4) did not encompass the top-ranked models from the main leaderboard. We entirely agree that evaluating the "Video Gain" for these models is crucial for strengthening the benchmark's framing.
All three strong models exhibit consistent positive video gains, losing 3.90 to 7.78 points when demonstration videos are removed. As shown in Table d, removing the video reduces scores by -4.88 points for Gemini-3.1-Pro, -3.90 points for GPT-5.6-sol, and -7.78 points for DeepSeek-V4.1-Flash-1017 . Furthermore, in Table e, the 2nd-ranked Gemini-3.6-Flash achieves a +4.69-point video gain under base rules. Together, these results show that stronger models possess superior ICL capabilities and consistently benefit from video demonstrations in V-ICAL. We report these comparisons in Appendix G.1 (Table 27) and summarize them in Section 4.4 and the Conclusion of the revised manuscript.
Table d : Controlled Comparison of Strong Models with (Baseline) and without Demonstration Videos
| Model | Baseline S [95% CI] | No-Video S [95% CI] | Δ No-Video [95% CI] |
|---|---|---|---|
| GPT-5.6-sol | 48.01 |
|44.11
|**-3.90
**| |Gemini-3.1-Pro|45.94
|41.06
|**-4.88
**| |DeepSeek-V4.1-Flash-1017†|44.10
|36.32
|**-7.78
**|
Note: S denotes the 0–100 normalized score; Δ No-Video = No-Video S - Baseline S. Its negation, -Δ No-Video, represents the Video Gain you emphasized, ranging from +3.90 to +7.78 points. †DeepSeek-V4.1-Flash-1017 is the latest updated version officially provided by DeepSeek during the rebuttal period; both Baseline and No-Video settings were evaluated uniformly on this latest version.
Table e : Original Table 4 (Full 37 Environments, 342 Tasks) Augmented with 95% Bootstrap Confidence Intervals
| Model | Base Rule + Video | Base Rule + No Video | Detailed Rule + Video | Detailed Rule + No Video |
|---|---|---|---|---|
| Gemini-3.6-Flash | 49.45 |
|44.76
|48.35
|46.76
| |Gemini-3-Flash|39.60
|41.16
|44.10
|43.62
| |Seed-2.0-Lite|36.98
|29.92
|37.15
|37.18
| |Qwen3.5-397B-A17B|32.54
|30.67
|33.87
|34.16
|
All three strong models exhibit consistent positive video gains, losing 3.90 to 7.78 points when demonstration videos are removed. As shown in Table d, removing the video reduces scores by -4.88 points for Gemini-3.1-Pro, -3.90 points for GPT-5.6-sol, and -7.78 points for DeepSeek-V4.1-Flash-1017 . Furthermore, in Table e, the 2nd-ranked Gemini-3.6-Flash achieves a +4.69-point video gain under base rules. Together, these results show that stronger models possess superior ICL capabilities and consistently benefit from video demonstrations in V-ICAL. We report these comparisons in Appendix G.1 (Table 27) and summarize them in Section 4.4 and the Conclusion of the revised manuscript.
2. Name-Masked Controlled Ablation for Strong Models (Response to W1, Q1 Part 2)
W1 & Q1 (Part 2): ...and the prompt gives the game name directly App\. B\.3. Many of these are well-known games such as Pong and Tetris, so the models' prior knowledge may play a large role. Given this, the leaderboard in Table 1 seems to measure game-playing from priors and rules more than the ability suggested by the title and abstract. Could you report results for the top-ranked models without video and with the game name masked...
In response to your thoughtful question regarding whether providing the exact {game_name} might inadvertently allow models to rely primarily on prior knowledge, we conducted a controlled ablation study on the same 21 representative environments (165 tasks) for the three major closed-source strong models. In this ablation, we replaced the {game_name} in the prompt with an anonymous placeholder .
Table f : Controlled Comparison of Strong Models with Original (Baseline) and Masked (Masked) Environment Names
| Model | Baseline S | Masked S | Δ Masked |
|---|---|---|---|
| Gemini-3.6-Flash | 56.54 | 57.42 | +0.88 |
| GPT-5.6-sol | 48.01 | 47.98 | -0.03 |
| Gemini-3.1-Pro | 45.94 | 45.09 | -0.85 |
Note: Masked S is the normalized score after masking all instances of {game_name} in the prompt. Δ Masked = Masked S - Baseline S.
Masking environment names leaves normalized scores virtually unchanged \Delta\pm 0.88. As shown in Table f, score fluctuations across all three strong models stay well within 1 point after masking {game_name} . This confirms that model performance on V-ICAL is not driven by memorized priors tied to specific game titles.
We sincerely thank you for your highly positive evaluation of the V-ICAL research direction, the reproducibility of our benchmark construction, and our failure-attribution analysis validated through blind human verification. We deeply appreciate your encouraging remark that "none of these issues undermine the benchmark's value as a resource," and that incorporating no-video/name-masked ablations for strong models alongside confidence intervals would give you "more confidence to champion the paper." We have strictly followed your valuable suggestions, completed all requested additional experiments, and detailed our responses to your specific questions below.