acceptodds
Under review as a conference paper at ICLR 2027

V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments

Abstract

While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particularly regarding video demonstrations in interactive environments. For multimodal agents, learning from videos presents unique challenges: they must translate in-context demonstrations into executable policies, ground these policies in novel visual states, and iteratively refine actions based on environmental feedback. We introduce V-ICAL, a novel benchmark designed to evaluate video-based ICL for multimodal agents. Comprising 342 interactive tasks across 37 environments, V-ICAL utilizes human-curated demonstration videos as task-specific behavioral exemplars, evaluating agents through sustained interaction from a target initialization. The benchmark seamlessly connects in-context knowledge induction with core agentic capabilities, including state grounding, temporal memory, planning, and adaptation in dynamic environments. Extensive evaluations across 19 state-of-the-art multimodal agents reveal significant limitations: the best-performing model, Seed-2.1-Pro, achieves a score of only 54.4/100, while other leading models (e.g., Gemini-3.1-Pro, Gemini-3.6-Flash, GPT-5.6-sol) fail to surpass 50, far below the human baseline of 83.6. Controlled comparisons further demonstrate that current agents struggle to reliably translate video exemplars into effective policies, failing to yield consistent performance gains. Ultimately, V-ICAL exposes a critical gap in the ICL capabilities of multimodal agents, underscoring an urgent need for future research.

Primary area:
datasets and benchmarks
Keywords:
MLLM, Benchmark, Agent, Video

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vJiEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces V-ICAL, a benchmark that tests whether multimodal agents can learn task rules and strategies from video demonstrations and use them in closed-loop interaction. It contains 342 tasks built from 37 Gym environments. For each task, the agent receives 1–4 captioned demonstration clips, minimal rule text and the action space, and then interacts step by step from pixels. The authors evaluate 19 models along with random and human baselines; the best model, Seed-2.1-Pro, scores 54.4, against 83.6 for humans. An LLM auditor attributes failures with a four-stage model, with 95.0% agreement in human verification. Finally, a video/rule ablation shows that the gains from video are inconsistent.

Critical strengths
  1. The direction is a valuable one. The paper brings video in-context learning from offline QA into closed-loop interactive evaluation, filling a gap in how this ability is evaluated. The finding that current models cannot yet reliably benefit from video demonstrations is itself useful, as it points to a clear open problem.
  2. The benchmark is carefully built and reproducible. Environments are screened with explicit criteria, start states come from human play and reach decision-heavy situations rarely seen at reset, scoring is calibrated per task, and the authors commit to releasing raw trajectories. As video understanding moves from offline QA toward interactive agents, a closed-loop testbed like this is of real value to the community.
  3. The failure analysis goes beyond a single score and is verified by blind human checks. The finding that failures come mostly from state understanding and planning rather than action execution is useful for model developers.
Critical weaknesses

W1. The paper has not shown that the benchmark measures video in-context learning. The authors' own ablation (§4.4, Table 4) shows that removing the videos changes scores only slightly and not in a consistent direction; for example, under base rules Gemini-3-Flash scores higher without video (41.2 vs 39.6), and with detailed rules the video/no-video gap is within 1.6 points for all four models. The Conclusion also states that demonstrations "yield no consistent aggregate gains". In addition, the ablation does not include the top-ranked models, and the prompt gives the game name directly (App. B.3). Many of these are well-known games such as Pong and Tetris, so the models' prior knowledge may play a large role. Given this, the leaderboard in Table 1 seems to measure game-playing from priors and rules more than the ability suggested by the title and abstract.

W2. The main results lack uncertainty estimates. Each model is run once per task, and neither Table 1 nor Table 4 reports confidence intervals, while many ranking gaps and ablation differences are only 1–2 points and may fall within noise. The two main memory/dynamics conclusions face the same issue: Single-memory has only 4 environments and Static only 10, and they largely correspond to particular environment families, so it is currently hard to separate these conclusions from the difficulty of the environments themselves.

None of these issues undermines the value of the benchmark as a resource. If the authors can add no-video/name-masked controls for the strong models with confidence intervals, I would be more confident in supporting the paper.

Questions for authors
  1. Could you report results for the top-ranked models without video and with the game name masked, and use the "video gain" (with video minus without video) as a primary metric? A consistent, significant gain for the strong models would make the benchmark's framing much more convincing; if there is none, the paper might be reframed around the finding that current models cannot yet make effective use of video demonstrations.
  2. Could you add environment-clustered bootstrap confidence intervals to Tables 1 and 4, and report each model's decoding settings (temperature, reasoning budget, etc.)?
  3. Could the memory/dynamics conclusions be tested within the same environment family to rule out environment difficulty?
  4. The prompt tells the model that the demonstrations are "not necessarily good" play and are only for understanding environment dynamics (App. B.3). Could this affect the strategy-transfer results in Table 2? Did you try a prompt that encourages imitating strategies?
7 comments

We sincerely thank you for your highly positive evaluation of the V-ICAL research direction, the reproducibility of our benchmark construction, and our failure-attribution analysis validated through blind human verification. We deeply appreciate your encouraging remark that "none of these issues undermine the benchmark's value as a resource," and that incorporating no-video/name-masked ablations for strong models alongside confidence intervals would give you "more confidence to champion the paper." We have strictly followed your valuable suggestions, completed all requested additional experiments, and detailed our responses to your specific questions below.

1. No-Video Controlled Ablation & Video Gain Analysis for Strong Models (Response to W1, Q1 Part 1)

W1: The paper has not shown that the benchmark measures video in-context learning. The authors' own ablation §44˙,Table4§4\.4, Table 4 shows that removing the videos changes scores only slightly and not in a consistent direction; for example, under base rules Gemini-3-Flash scores higher without video 412˙vs396˙41\.2 vs 39\.6, and with detailed rules the video/no-video gap is within 1.6 points for all four models. The Conclusion also states that demonstrations "yield no consistent aggregate gains". In addition, the ablation does not include the top-ranked models, ...

Q1: Could you report results for the top-ranked models without video and with the game name masked, and use the "video gain" (with video minus without video) as a primary metric? A consistent, significant gain for the strong models would make the benchmark's framing much more convincing; if there is none, the paper might be reframed around the finding that current models cannot yet make effective use of video demonstrations.

We appreciate your insightful observation in W1 and Q1 that our initial video/no-video ablation (Table 4) did not encompass the top-ranked models from the main leaderboard. We entirely agree that evaluating the "Video Gain" for these models is crucial for strengthening the benchmark's framing.

All three strong models exhibit consistent positive video gains, losing 3.90 to 7.78 points when demonstration videos are removed. As shown in Table d, removing the video reduces scores by -4.88 points for Gemini-3.1-Pro, -3.90 points for GPT-5.6-sol, and -7.78 points for DeepSeek-V4.1-Flash-1017 correspondingtopositivevideogainsof+39˙0to+77˙8corresponding to positive video gains of +3\.90 to +7\.78. Furthermore, in Table e, the 2nd-ranked Gemini-3.6-Flash achieves a +4.69-point video gain under base rules. Together, these results show that stronger models possess superior ICL capabilities and consistently benefit from video demonstrations in V-ICAL. We report these comparisons in Appendix G.1 (Table 27) and summarize them in Section 4.4 and the Conclusion of the revised manuscript.

Table d ResponsetoW1&Q1Part1Response to W1 \& Q1 Part 1: Controlled Comparison of Strong Models with (Baseline) and without No−VideoNo-Video Demonstration Videos

ModelBaseline S [95% CI]No-Video S [95% CI]Δ No-Video [95% CI]
GPT-5.6-sol48.01
350˙2,615˙435\.02, 61\.54

|44.11

308˙4,579˙230\.84, 57\.92

|**-3.90

−119˙0,40˙5-11\.90, 4\.05

**| |Gemini-3.1-Pro|45.94

347˙5,579˙134\.75, 57\.91

|41.06

292˙4,534˙529\.24, 53\.45

|**-4.88

−126˙7,25˙2-12\.67, 2\.52

**| |DeepSeek-V4.1-Flash-1017†|44.10

326˙4,562˙532\.64, 56\.25

|36.32

250˙7,487˙625\.07, 48\.76

|**-7.78

−148˙4,−15˙1-14\.84, -1\.51

**|

Note: S denotes the 0–100 normalized score; Δ No-Video = No-Video S - Baseline S. Its negation, -Δ No-Video, represents the Video Gain you emphasized, ranging from +3.90 to +7.78 points. †DeepSeek-V4.1-Flash-1017 is the latest updated version officially provided by DeepSeek during the rebuttal period; both Baseline and No-Video settings were evaluated uniformly on this latest version.

Table e ResponsetoW1,W2&Q2Response to W1, W2 \& Q2: Original Table 4 (Full 37 Environments, 342 Tasks) Augmented with 95% Bootstrap Confidence Intervals

ModelBase Rule + VideoBase Rule + No VideoDetailed Rule + VideoDetailed Rule + No Video
Gemini-3.6-Flash49.45
399˙3,591˙139\.93, 59\.11

|44.76

358˙4,539˙435\.84, 53\.94

|48.35

395˙6,572˙739\.56, 57\.27

|46.76

374˙1,563˙537\.41, 56\.35

| |Gemini-3-Flash|39.60

312˙7,484˙431\.27, 48\.44

|41.16

329˙6,495˙832\.96, 49\.58

|44.10

356˙4,529˙735\.64, 52\.97

|43.62

348˙1,527˙834\.81, 52\.78

| |Seed-2.0-Lite|36.98

290˙9,454˙329\.09, 45\.43

|29.92

224˙1,381˙722\.41, 38\.17

|37.15

291˙7,456˙429\.17, 45\.64

|37.18

287˙0,459˙828\.70, 45\.98

| |Qwen3.5-397B-A17B|32.54

245˙8,411˙724\.58, 41\.17

|30.67

234˙0,385˙223\.40, 38\.52

|33.87

263˙4,419˙126\.34, 41\.91

|34.16

259˙5,429˙825\.95, 42\.98

|

All three strong models exhibit consistent positive video gains, losing 3.90 to 7.78 points when demonstration videos are removed. As shown in Table d, removing the video reduces scores by -4.88 points for Gemini-3.1-Pro, -3.90 points for GPT-5.6-sol, and -7.78 points for DeepSeek-V4.1-Flash-1017 correspondingtopositivevideogainsof+39˙0to+77˙8corresponding to positive video gains of +3\.90 to +7\.78. Furthermore, in Table e, the 2nd-ranked Gemini-3.6-Flash achieves a +4.69-point video gain under base rules. Together, these results show that stronger models possess superior ICL capabilities and consistently benefit from video demonstrations in V-ICAL. We report these comparisons in Appendix G.1 (Table 27) and summarize them in Section 4.4 and the Conclusion of the revised manuscript.

2. Name-Masked Controlled Ablation for Strong Models (Response to W1, Q1 Part 2)

W1 & Q1 (Part 2): ...and the prompt gives the game name directly App\. B\.3. Many of these are well-known games such as Pong and Tetris, so the models' prior knowledge may play a large role. Given this, the leaderboard in Table 1 seems to measure game-playing from priors and rules more than the ability suggested by the title and abstract. Could you report results for the top-ranked models without video and with the game name masked...

In response to your thoughtful question regarding whether providing the exact {game_name} eg˙,˙Pong,Tetrise\.g\., Pong, Tetris might inadvertently allow models to rely primarily on prior knowledge, we conducted a controlled ablation study on the same 21 representative environments (165 tasks) for the three major closed-source strong models. In this ablation, we replaced the {game_name} in the prompt with an anonymous placeholder MaskedS,see∗∗Tablef∗∗Masked S, see **Table f**.

Table f ResponsetoW1&Q1Part2Response to W1 \& Q1 Part 2: Controlled Comparison of Strong Models with Original (Baseline) and Masked (Masked) Environment Names

ModelBaseline SMasked SΔ Masked
Gemini-3.6-Flash56.5457.42+0.88
GPT-5.6-sol48.0147.98-0.03
Gemini-3.1-Pro45.9445.09-0.85

Note: Masked S is the normalized score after masking all instances of {game_name} in the prompt. Δ Masked = Masked S - Baseline S.

Masking environment names leaves normalized scores virtually unchanged ∗∗**\Delta∗∗Maskedwithin∗∗** Masked within **\pm 0.88∗∗points** points. As shown in Table f, score fluctuations across all three strong models stay well within 1 point after masking {game_name} withGPT−56˙−soldifferingbyonly∗∗−00˙3points∗∗with GPT-5\.6-sol differing by only **-0\.03 points**. This confirms that model performance on V-ICAL is not driven by memorized priors tied to specific game titles.

Reviewer_aeCVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces V-ICAL Bench, a suite of existing interactive tasks repurposed on evaluating the in-context learning capabilities from video actions for foundation models. The authors curated 342 tasks across 37 environments and evaluated 19 proprietary and open-weight models, and found that even the strongest model seed 2.1 pro lags way behind human. The authors did fine-grained analysis categorizing different kinds of failure mode and how often different model makes mistakes from these categories as well as ablating different part of the context supplied to the model such as video and specificity of the rules and ultimately how they contribute to model performance.

Critical strengths
  • I believe the evaluation is very well motivated and timely. The capability to learn from video-action context has implications to many real-world downstream domains, such as robotics and computer-using agents. Although the nature of all tasks is based on synthetic renderings and simulations, I believe they still serve as very useful proxies to evaluate and diagnose such capabilities in a controlled approach.
  • The evaluation scope is impressively comprehensive. Proprietary models of different providers and different generations are tested, and lots of frontier and affordable open-weight models are also tested. This strengthens the concrete gap between model and human capabilities on these specified tasks.
  • The paper is very well written and easy to follow and digest.
Critical weaknesses
  • My biggest concern is that the numbers in main table do not reliably reflect what the authors intended to measure, ICL capabilities from interactive data/environments. Many of these models could have been pre-trained or mid-trained on the environments that the authors evaluate the models with, so the numbers confound ICL capabilities with task leakage.
  • The framing is moderately misleading. The title, abstract and introduction suggest V-ICAL Bench as a "video" ICL capability benchmark, while the actual ICL examples not only contain continuous frames but also the actions that lead to these frames. This is a minor point, but the current setup is better framed as a visual interaction ICL benchmark rather than video ICL benchmark.
  • Because all the environments appear synthetic, I really hoped that the authors could leverage this characteristic and generate different kind of training data to learn how one might improve interactive ICL capabilities for the models. This could be a very nice-to-have study.
  • The related work could have been more comprehensive. Multiple prior works such as VisGym, VisualAgentBench and VAGEN are relevant studies of multimodal agents in interactive environments and share lots of similar findings. I would encourage the authors to contextualize these findings relative to prior work and clarify what additional insights that emerge specifically in the ICL aspect of multimodal interaction.
Questions for authors
  • If the goal is to measure multimodal interactive ICL capabilities, I think the main table should be entirely swapped with Table 4, where one measures the performance gap with and without the existence of multimodal interactive examples in the context. Could authors provide that for the whole main table? The current main table confounds ICL capabilities with base model performance which is misleading.
  • Could authors justify why the construction of the evaluation is more video-focused rather than multimodal interaction focused? In addition to my framing question, I think one useful ablation is to study the performance gap between frames+actions vs frames-only. This separates video understanding from interaction understanding.
10 comments

We sincerely thank Reviewer aeCV for the highly positive evaluation of V-ICAL's research motivation, evaluation breadth, and writing quality. To address the raised concerns and questions, we have carefully conducted the two requested experiments the with/without exemplar comparison for W1/Q1, and the Frames+Actions vs\. Frames-Only ablation for W2/Q2. We present the results in two separate tables and provide detailed point-by-point responses below.

1. Response to W1 & Q1: Investigating the Performance Gap With/Without Multimodal Interactive Exemplars (Expanding Table 4 to More Main Table Models)

W1: My biggest concern is that the numbers in main table do not reliably reflect what the authors intended to measure, ICL capabilities from interactive data/environments. Many of these models could have been pre-trained or mid-trained on the environments that the authors evaluate the models with, so the numbers confound ICL capabilities with task leakage.

Q1: If the goal is to measure multimodal interactive ICL capabilities, I think the main table should be entirely swapped with Table 4, where one measures the performance gap with and without the existence of multimodal interactive examples in the context. Could authors provide that for the whole main table? The current main table confounds ICL capabilities with base model performance which is misleading.

We appreciate the insightful observation regarding the potential confounding of ICL capabilities with pre-training task leakage in the main table. The suggestion to investigate the performance gap with and without multimodal interactive exemplars (as in Table 4) and extend it to more representative models is highly valuable.

First, we would like to clarify why we retained the current main table (Table 1) rather than entirely replacing it with the performance difference ∗∗**\Delta∗∗** between the with-video and no-video settings:

  1. The definition of interactive ICL capabilities extends beyond a simple subtraction of scores: In a multimodal interactive environment, ICL capability cannot be fully captured merely by the performance difference Δ\Delta. Rather, it reflects a model's comprehensive ability—under fixed parameters—to extract transferable task rules, action-state dynamics, and behavioral strategies from current interactive demonstrations. This capability involves binding these elements to new visual states and achieving task goals through a complete pipeline of state grounding, planning, and action commitment during continuous interaction.

  2. No-Video still involves online context adaptation during multi-turn closed-loop interactions, preventing a simple subtraction: When executing tasks, the model interacts within the same environment as the video demonstration across multiple turns. Even when the demonstration video is removed at step 0 (No-Video), as the model explores the environment from steps 1…T1 \dots T, it retains the sequence of interactive frames in its context window. It thereby acquires environmental dynamics equivalent to parts of the video demonstration from its own historical trajectory ie˙,˙test−timetrial−and−erroradaptationi\.e\., test-time trial-and-error adaptation. Consequently, ScoreVideo−ScoreNo-Video\text{Score}_{\text{Video}} - \text{Score}_{\text{No-Video}} strictly measures the "marginal contribution of external video demonstrations," whereas the main table score (with video) measures the holistic Agentic ICL capability, which encompasses both "video rule induction" and "interactive history contextual reasoning."

  3. Ranking purely by Δ\Delta would paradoxically penalize models with strong foundational perception and reasoning: Combining the original Table 4 with Table b below, the improvements for Seed-2.0-Lite Δ=+7.1\Delta = +7.1 and DeepSeek-V4.1-Flash-1017 Δ=+7.78\Delta = +7.78 are nearly double that of GPT-5.6-sol Δ=+3.90\Delta = +3.90. However, the latter's actual interactive score 480˙148\.01 is significantly higher than the former two 370˙and441˙0,respectively37\.0 and 44\.10, respectively. Furthermore, beyond just the with/without demonstration comparison, Section 4.3 (Table 2) of the original paper provides complementary trajectory-level evidence. We directly examined whether the models' visible reasoning and actual actions reflect rules and strategies shown in the video but not explicitly provided in the base text rules, thus diagnosing rule and strategy transfer consistent with the demonstration beyond what the final score alone can reveal.

Second, regarding the performance gap with/without video exemplars and the expansion to more main table models: Section 4.4 (Table 4) of the original paper already presented a controlled orthogonal ablation study of Video vs. No-Video across all 37 environments (342 tasks). Under the Base Rule, providing the video yielded significant improvements in 3 out of 4 models eg˙,˙+47˙forGemini−36˙−Flashand+71˙forSeed−20˙−Lite;weomitthepreviouslyreporteddataheree\.g\., +4\.7 for Gemini-3\.6-Flash and +7\.1 for Seed-2\.0-Lite; we omit the previously reported data here. To address the valuable suggestion of evaluating more main table models, we selected the top closed-source models GPT−56˙−sol,Gemini−31˙−ProGPT-5\.6-sol, Gemini-3\.1-Pro and the latest official iteration of the strongest open-weight model DeepSeek−V41˙−Flash−1017DeepSeek-V4\.1-Flash-1017. Due to API and time constraints, we conducted this supplementary comparison on a representative subset of 21 environments (165 tasks) covering 7 major environment families, comparing the settings with demonstration videos (Baseline S) and without No−VideoSNo-Video S.

Table b AddressingW1&Q1Addressing W1 \& Q1: Controlled Comparison of Representative Main Table Models With (Baseline) and Without No−VideoNo-Video Demonstration Videos

ModelBaseline SNo-Video SΔ No-Video
GPT-5.6-sol48.0144.11-3.90
Gemini-3.1-Pro45.9441.06-4.88
DeepSeek-V4.1-Flash-1017†44.1036.32-7.78

Note: S represents the 0–100 normalized score; Δ No-Video = No-Video S - Baseline S. †DeepSeek-V4.1-Flash-1017 is the latest version provided by DeepSeek during the rebuttal period. Both Baseline S 441˙044\.10 and No-Video S 363˙236\.32 on our subset were uniformly evaluated on this latest version.

Removing the demonstration video consistently lowers the normalized scores across all evaluated strong models ∗∗**\Delta∗∗No−Videorangingfrom−39˙0to−77˙8** No-Video ranging from -3\.90 to -7\.78. As shown in Table b, performance drops by -3.90 for GPT-5.6-sol, -4.88 for Gemini-3.1-Pro, and -7.78 for DeepSeek-V4.1-Flash-1017 when videos are removed. This demonstrates that strong models on V-ICAL rely significantly on multimodal interactive demonstrations in the context. We have formally integrated this ablation study into the revised manuscript.

2. Response to W2 & Q2: Terminology Refinement and Controlled Ablation of Frames+Actions vs. Frames-Only

W2: The framing is moderately misleading. The title, abstract and introduction suggest V-ICAL Bench as a "video" ICL capability benchmark, while the actual ICL examples not only contain continuous frames but also the actions that lead to these frames. This is a minor point, but the current setup is better framed as a visual interaction ICL benchmark rather than video ICL benchmark.

Q2: Could authors justify why the construction of the evaluation is more video-focused rather than multimodal interaction focused? In addition to my framing question, I think one useful ablation is to study the performance gap between frames+actions vs frames-only. This separates video understanding from interaction understanding.

We completely agree with the insightful observation regarding the terminology and the composition of the ICL exemplars.

First, regarding the terminology refinement W2W2: We sincerely appreciate this correction. In the revised manuscript, we will adjust our phrasing to precisely define V-ICAL as an In-Context Learning evaluation benchmark based on "interactive video demonstrations with action annotations," better reflecting its multimodal interaction nature.

Reviewer_SK2QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces V-ICAL, a benchmark for evaluating whether multimodal agents can learn task-specific rules and strategies from video demonstrations and apply them through closed-loop interaction. The benchmark contains 342 task configurations across 37 interactive environments, with human-curated demonstrations and evaluation based on task completion and normalized progress scores. The authors evaluate 19 multimodal models against human and random baselines. The paper further introduces a four-stage trajectory-auditing framework covering knowledge induction, state grounding, planning, and action commitment. This analysis is supported by human verification of automated labels.

Critical strengths
  1. A valuable benchmark targeting an underexplored capability.

  2. Comprehensive empirical evaluation and informative diagnostic analysis.

Critical weaknesses
  1. Limited isolation of video in-context learning from general agent capabilities and prior knowledge. Although the benchmark is designed to evaluate video ICL, it remains unclear how much observed performance actually depends on learning from the demonstrations.

  2. Confounding factors weaken the interpretation of memory and environment-dynamics bottlenecks. The paper attributes substantial performance degradation to temporal memory requirements and dynamic environments. However, these properties are defined largely at the environment level rather than being independently manipulated.

Questions for authors
  1. Could the authors provide video versus no-video comparisons for the strongest models, particularly Seed-2.1-Pro and GPT-5.6-sol, ideally with per-environment results and uncertainty estimates?

  2. To what extent do the reported performance gaps remain after controlling for environment family, task difficulty, interaction horizon, and the overlap between memory requirements and environment dynamics?

2 comments

We sincerely thank the reviewer for acknowledging the research value of V-ICAL, our comprehensive empirical evaluation, and the trajectory-auditing framework built around four decision-making stages and validated through blind human verification. Below, we address your concerns and questions point-by-point, providing dedicated tables for clarity.

1. Controlled Comparison of Strong Models Video vs\. No-Video AddressingW1&Q1Addressing W1 \& Q1

W1: Limited isolation of video in-context learning from general agent capabilities and prior knowledge. Although the benchmark is designed to evaluate video ICL, it remains unclear how much observed performance actually depends on learning from the demonstrations.

Q1: Could the authors provide video versus no-video comparisons for the strongest models, particularly Seed-2.1-Pro and GPT-5.6-sol, ideally with per-environment results and uncertainty estimates?

We appreciate your concern regarding the extent to which the observed performance relies on learning from demonstration videos versus the models' inherent priors and general capabilities.

First, as presented in Section 4.4 (Table 4) of our original manuscript, we have already conducted a two-factor controlled ablation study *Video vs\. No Video* × *Base vs\. Detailed Rule* across all 37 environments. Under the basic rule setting, providing video demonstrations yielded a +7.1 improvement for Seed-2.0-Lite and +4.7 for Gemini-3.6-Flash (please refer to the original paper for detailed metrics).

To further address your specific request for stronger models, we evaluated a representative subset of 21 environments (comprising 165 tasks and covering all 7 environment families) to balance computational costs and time constraints during the rebuttal period. We tested top-tier closed-source models GPT−56˙−sol,Gemini−31˙−ProGPT-5\.6-sol, Gemini-3\.1-Pro and a leading open-weight model DeepSeek−V41˙−Flash−1017DeepSeek-V4\.1-Flash-1017 under "Baseline S" (with video) and "No-Video S" settings.

Table a AddressingW1&Q1Addressing W1 \& Q1: Controlled Video vs. No-Video Comparison for Strong Models

ModelBaseline S [95% CI]No-Video S [95% CI]Δ No-Video [95% CI]
GPT-5.6-sol48.01
350˙2,615˙435\.02, 61\.54

|44.11

308˙4,579˙230\.84, 57\.92

|-3.90

−119˙0,40˙5-11\.90, 4\.05

| |Gemini-3.1-Pro|45.94

347˙5,579˙134\.75, 57\.91

|41.06

292˙4,534˙529\.24, 53\.45

|-4.88

−126˙7,25˙2-12\.67, 2\.52

| |DeepSeek-V4.1-Flash-1017†|44.10

326˙4,562˙532\.64, 56\.25

|36.32

250˙7,487˙625\.07, 48\.76

|-7.78

−148˙4,−15˙1-14\.84, -1\.51

|

Note: S denotes the 0–100 normalized score. Δ No-Video = No-Video S - Baseline S. A negative value indicates a performance drop after removing the demonstration video. †DeepSeek-V4.1-Flash-1017 is the latest version provided officially during the rebuttal period; both Baseline S and No-Video S were evaluated on this identical latest version.

Removing demonstration videos consistently degrades performance across all three strong models. As shown in Table a, scores drop by -3.90 for GPT-5.6-sol, -4.88 for Gemini-3.1-Pro, and -7.78 for DeepSeek-V4.1-Flash-1017. This empirically confirms that strong models substantively leverage contextual information from demonstration videos during closed-loop interactions, rather than relying solely on zero-shot priors or general capabilities.

2. Clarifications on Memory Requirement, Environment Dynamics, and Confounding Factors AddressingW2&Q2Addressing W2 \& Q2

W2: Confounding factors weaken the interpretation of memory and environment-dynamics bottlenecks. The paper attributes substantial performance degradation to temporal memory requirements and dynamic environments. However, these properties are defined largely at the environment level rather than being independently manipulated.

Q2: To what extent do the reported performance gaps remain after controlling for environment family, task difficulty, interaction horizon, and the overlap between memory requirements and environment dynamics?

We sincerely thank you for this rigorous observation. You rightly point out that "Memory Requirement" and "Environment Dynamics" are primarily defined at the environment level, rather than being isolated through intervention-based ablations within the same task.

To clarify, in V-ICAL, the categorizations in our main table—Single-frame (Single) vs. Multi-frame (Multiple) MemoryRequirement:whetherthecurrentframeissufficientforactionselectionorhistoricalobservationsmustbeintegrated,eg˙,˙partialobservabilityin∗CustomMultiRoom∗orvelocityestimationin∗FlappyBird∗Memory Requirement: whether the current frame is sufficient for action selection or historical observations must be integrated, e\.g\., partial observability in *CustomMultiRoom* or velocity estimation in *FlappyBird* and Static vs. Dynamic Environment Dynamics: whether the environment changes only in response to agent actions vs\. evolves autonomously over time—serve as diagnostic classifications based on the objective properties of the tasks and environments.

During our evaluation, the history retention mechanisms and prompts were kept strictly identical across all models; no evaluation variables were altered. We wish to highlight that under this objective classification, nearly all evaluated models systematically underperformed on the Multi-frame / Dynamic categories compared to the Single-frame / Static categories. Notably, the Random baseline exhibited the exact opposite trend, which helps rule out biases stemming merely from environmental scoring scales.

However, we fully agree with your assessment: our unified evaluation protocol inherently cannot rule out confounding factors such as environment family, inherent task difficulty, interaction horizon, and the intersection between memory requirements and dynamics. Because our current results reflect descriptive subgroup comparisons based on objective task properties rather than intervention-based ablations, we will revise Section 4.2 to sharpen these classification definitions and explicitly tone down causal attribution claims in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.