acceptodds
Under review as a conference paper at ICLR 2027

Self-Improving Video-to-Action Models

Abstract

Video generative models are an emerging tool for embodied agents: they can serve as scalable visual planners, generating videos that depict how to complete previously unseen tasks. However, executing these plans typically requires a video-to-action or "inverse dynamics" model trained on large, human-collected video-action datasets, severely limiting scalability. We introduce Mirror, a self-supervised approach that learns video-to-action mappings through environment interactions, without prior action data or hand-crafted rewards. Mirror learns by attempting to follow a visual plan by executing its best guess of the underlying actions in the environment. Mirror then trains on the resulting video-action pairs, incrementally self-improving and following the visual planner more and more closely. Across Atari, OGBench, and DeepMind Control, Mirror achieves overall performance comparable to leading learning-from-observation methods with learned video plans, and near-expert performance with oracle plans. On Minecraft BASALT and text-conditioned tasks, it outperforms methods with access to expert actions or hand-engineered task knowledge. Using generated video plans, it surpasses VPT-BC in survival mode, even though VPT-BC was trained on thousands of hours of action-labeled human gameplay. With oracle video plans, it learns to acquire diamonds without task rewards, using an order of magnitude fewer environment interactions than DreamerV3. Our results suggest that advances in video models can translate into increasingly capable agents without a corresponding need for hand-crafted rewards or large collections of expert action data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
video planning, inverse dynamics, self-supervised learning, learning from observation, embodied agents, imitation learning, exploration, video diffusion model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Kev5OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to learn an inverse dynamics model (called a V2A model) from an expert visual planner through self-collected experience. Rather than relying on random exploration, the visual planner is utilized to keep replanning meaningful rollouts to bring the agent back to the task-specific stuck points until the inverse dynamics model outputs a correct action to overcome it. The inverse dynamics model can also structurally learn from arbitrary quality experience. The authors demonstrate their method for learning IDMs on a variety of simulated environments such as Minecraft, DMC, OGBench, and Atari.

Critical strengths

The strength of this paper is its clarity of writing, and its straightforward approach. Furthermore, its findings support the scaling of video modeling efforts, showing how they can facilitate robotic decision-making through visual planning.

Critical weaknesses

One weakness of this work is its assumed access to an expert video generative model. This should not be assumed to be easy to get, for arbitrary practical robotic settings. Different robot embodiments may have different visual viewpoints, and adapting expertness into the video model correspondingly may turn into an expensive and adhoc endeavour.

Another weakness of this work is its contextualization with prior work, which has investigated self-improvement for visual planners. In particular, SILVR [1] performs self-improvement on both the video model as well as the IDM, whereas this proposed work only self-improves the IDM. SILVR also found sample efficiency benefits in self-improvement through visual planning, overlapping with the findings and takeaways of this work; but unlike this work, SILVR does not expect access to an expert video generative model (simply one that has adapted some level of environment visuals and dynamics [2]). This seems a more general setting - how quickly can a complete visual planner adapt to a new task, for which we do not assume expert behavior a priori, versus assuming a video model already encodes successful behavior within it. Similar prior work also look into self-improvement for WAMs, which can be thought of a singular model that jointly predicts video and actions (rather than the hierarchical approach of a video model and IDM) - it would be good to highlight efforts to update WAMs with respect to online rollout experience as well [3].

Another weakness is its evaluation suite. Particularly for synthetic environments, an argument could be made that interaction data is cheap and easy to collect - particularly in the age of agents that can spin up virtual environments and perform arbitrary interactions at scale. IDMs (or V2A models as the authors call them) can be trained on arbitrary quality interaction data, so such scaled generation data gathered from synthetic environments should result in a strong IDM up front - studying how they can be self-improved from scratch seems rather contrived for synthetic environments. Rather, it is learning how to achieve an expert-level video model that is interesting for synthetic environments, as this cannot be obtained from agents randomly interacting in synthetic environments at scale - but this work assumes access to an expert video model a priori. Thus, by only using synthetic environments, the evaluation setup of this work seems contrived. On the other hand, a convincing argument could be made for evaluating Mirror on real-world environments. In real-world settings, potentially different robots have different embodiments and action spaces - and a high-quality IDM may not necessarily be assumed a priori. A real-world experiment which takes an expert-level video model for a particular task, and reuses it for a new embodiment and action space (but, for example, keeps the camera angle the same for transferability), would be interesting and compelling evaluation, as agents cannot (currently) generate scaled interaction data in the real world for arbitrary embodiments. Thus, this reviewer believes this work would benefit from real-world experiments to be convincing.

Furthermore, as a small additional note on the synthetic experiment suite: as visual planners are expensive during inference, their benefit mostly lies in their high-level planning ability. However, apart from Minecraft, the other benchmark tasks such as OGBench do not particularly seem to benefit strongly from high-level visual planning.

[1] Luo et al., Self-Improving Loops for Visual Robotic Planning, ICLR 2026.

[2] Luo et al., Solving New Tasks by Adapting Internet Video Knowledge, ICLR 2025.

[3] Kim et al., Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning, arXiv 2026.

Questions for authors

In terms of replanning capabilities, how can the video model consistently guide the agent towards milestones it has failed on? Does this not require an even stronger video model than one that only encodes expert behavior - but a video model that can even perform recovery from failure or random states? If so, is this not a prohibitively expensive requirement to make of a video model for an arbitrary environment of interest (e.g. perfect expert recovery)?

8 comments
AuthorsOpenReview ↗

Official Response by Authors (1/2)

We thank the reviewer for the feedback and for the pointers to related work. In brief:

  • The revision now cites and discusses SILVR and the other works the reviewer points to (Concern 1). SILVR improves the video planner and Mirror learns the V2A model, so the two are complementary. Mirror also needs no success signal, so it applies to tasks whose success is hard to verify, such as BASALT.
  • New result: in Minecraft, a capable VP can be trained (Concern 2). We retrained our generative VP exclusively on human gameplay video in the revision. With this VP, Mirror now sets a new state of the art on the Minecraft Diamond Challenge. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.
  • Random interaction is not enough to train the V2A model on long-horizon tasks (Concern 3). Long horizons, which random interaction cannot cover, remain largely unsolved. This is why we evaluate on them and not on short-horizon real-world tasks, which are already within reach (Concern 4).

[Concern 1] Contextualization with SILVR and prior work

In particular, SILVR [1] performs self-improvement on both the video model as well as the IDM, whereas this proposed work only self-improves the IDM.

The revised related work now discusses all three works the reviewer points to, along with other prior work. We added SILVR (Luo et al., 2026), the video adaptation work of Luo et al. (2025), and Cosmos Policy (Kim et al., 2026), as well as VideoAgent, BCO, GCSL, and AnyPos.

We also discuss combining Mirror with SILVR as future work. In particular, we clarify how Mirror relates to SILVR:

  • Mirror does not need a verifiable success signal. SILVR filters rollouts by success, judged by ground truth, humans, or a VLM. For Mirror, every transition is a valid inverse dynamics label whether or not the attempt succeeded. It therefore applies to tasks whose success is hard to verify, such as BASALT.
  • SILVR and Mirror are complementary. SILVR improves the video planner from experience, while Mirror keeps the VP fixed and learns its V2A model from scratch. Where success can be verified, the two can be combined.

[Concern 2] Assumed access to an expert video model

One weakness of this work is its assumed access to an expert video generative model.

Our revised VP involves no expert policy. In the revision, we retrained the generative VP exclusively on human gameplay video, without actions or rewards. With it, Mirror sets a new state of the art on the hour-long Diamond Challenge: it crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3, and obtains diamonds in 5.5% of episodes in Dreamer 4's offline setting, against 0.7% for Dreamer 4. A VP trained only on human video thus goes surprisingly far. We agree that obtaining such a VP is not easy, which is why our VP architecture and training recipe are important contributions.

Capable VPs are becoming available. At short horizons in the real world, LVP (Chen et al., 2025) and VERA (Li et al., 2026) show that capable VPs can be trained from Internet-scale video and executed on real robots. Our results suggest that the same is possible at much longer horizons.

Does this not require an even stronger video model than one that only encodes expert behavior - but a video model that can even perform recovery from failure or random states?

Human gameplay already teaches the VP to recover. No additional data or supervision is needed. Human gameplay is full of mistakes and recoveries, so a VP trained on it learns to recover from failure states. The VP also replans from the agent's actual observation, so it always plans from wherever the agent has ended up. Videos of replanning and recovery are on our anonymous webpage (https://selfimprovingv2a.github.io/).

AuthorsOpenReview ↗

Official Response by Authors (2/2)

[Concern 3] IDMs are easy to train from arbitrary-quality interaction in synthetic environments

IDMs (or V2A models as the authors call them) can be trained on arbitrary quality interaction data, so such scaled generation data gathered from synthetic environments should result in a strong IDM up front.

In long-horizon tasks, random interactions never achieve sufficient coverage. In short-horizon tasks, random interactions can provide sufficient coverage to train an IDM. In long-horizon tasks, however, they cannot.

  • The action space is enormous. At each of 20 steps per second, the agent chooses among about half a million actions (≈5.2×105\approx 5.2 \times 10^5; Baker et al., 2022). A 20-minute episode thus admits more than 10137,00010^{137,000} action sequences, which can never be covered with random actions. Even human players trying to craft a diamond pickaxe obtain diamonds within 10 minutes in only 15% of episodes (Baker et al., 2022).
  • Random actions practically never break a single block. In Minecraft, breaking a block requires holding the attack button for about 60 consecutive steps, which a policy pressing attack with probability 0.5 does with probability about 10−1810^{-18}. This is why DreamerV3 relies on accelerated mining and simplified crafting, whereas Mirror learns mining from the VP's guidance alone.
  • Our experiments confirm this. Random-V2A, which is Mirror's V2A model trained only on random interaction, reaches 227 return on DMC against 788 for Mirror, and 0.38 success on OGBench against 0.93, both with the oracle VP. In text-conditioned Minecraft, a V2A model trained on random walks never chops a tree, while Mirror succeeds in 100% of episodes.

[Concern 4] Real-world experiments

Thus, this reviewer believes this work would benefit from real-world experiments to be convincing.

We focus on simulation because we target long horizons. We agree that real-world experiments would be valuable. However, running real robots on long-horizon tasks, where video-guided exploration matters most, is currently difficult at academic scale. Our self-improvement loop runs for 140 hours of interaction. Data is also a problem: we know of no public real-robot video data that has the horizon and diversity of human Minecraft gameplay, with episodes of up to 20 minutes that include many natural mistakes and recovery behaviors.

We also highlight that for short-horizon real-world tabletop settings, VERA (Li et al., 2026) has already shown that a frozen VP paired with an IDM trained on self-play data works well on real robots. This holds even for a 16-DoF Allegro hand, a contact-rich dexterous setting that usually requires large-scale RL in simulation. Thus, short horizons are already within reach, and long horizons remain the open problem. The difficulty comes from the horizon, not from whether the environment is synthetic or real. This is why we focus on long horizons.

[Concern 5] Benefit of visual planning on OGBench

Apart from Minecraft, the other benchmark tasks such as OGBench do not particularly seem to benefit strongly from high-level visual planning.

We agree that the control benchmarks test action grounding, not planning. This is by design. Mirror's primary purpose is to turn the prior of large pretrained video models into actions, and the control benchmarks isolate this step. There, given a capable VP, we test whether Mirror grounds it into actions better than standard learning-from-observation baselines such as LAIfO and VIPER. We demonstrate the benefit of long-horizon visual planning in Minecraft.

We also note the limits of these baselines. LAIfO learns by adversarially matching in-domain expert demonstrations online. It therefore cannot draw on a pretrained video model, and in our experience its training is unstable and sensitive to hyperparameters. VIPER does draw on a pretrained video model, but it is not entirely free from adversarial dynamics either, due to its exploration term. It is also prone to reward hacking, as video likelihoods become unreliable in high dimensions.

We hope this addresses the reviewer's concerns, and we would be grateful if the reviewer would consider updating the score.

Dear Reviewer Kev5,

As the discussion period draws to a close, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify any remaining points.

Thank you,
The Authors

Reviewer_5ScnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Mirror, which learns video-to-action mappings through environment interaction without expert action labels or task rewards. It alternates between executing video plans and training on the resulting observations and actions, allowing unsuccessful attempts to provide valid supervision. The authors also develop a capable video planner supporting efficient long-context, closed-loop planning. Experiments across Minecraft and 24 standard control tasks demonstrate strong performance, including diamond acquisition in Minecraft when guided by an oracle video planner.

Critical strengths
  1. The method is simple and well motivated, combining video-guided exploration with supervised learning from achieved transitions.
  2. Beyond the main learning algorithm, the paper develops a strong video planner with efficient long-context inference, text-conditioned control, and evidence of in-context adaptation.
  3. The experiments are extensive, with detailed analyses of planner quality, exploration, generalization, and the differences between learned and oracle planners.
  4. The paper is clearly written, with intuitive explanations and thorough implementation details.
Critical weaknesses

The main limitation is incomplete positioning relative to prior work on exploration-based inverse dynamics learning and hindsight-conditioned supervised learning. Discussing these connections would help distinguish the contribution of video-guided data collection from the underlying learning principle.

Questions for authors
  1. Beyond the existing Random-V2A comparisons, could you discuss Mirror relative to AnyPos, particularly when task-agnostic exploration provides sufficient coverage for action decoding?
  2. Could you clarify the relationship between Mirror’s training principle and hindsight-style method in GCSL?
4 comments
AuthorsOpenReview ↗

Official Response by Authors

We thank the reviewer for the positive assessment and the insightful questions. We also highlight a new result: Mirror now sets a new state of the art on the Minecraft Diamond Challenge. In the revision, we retrained the generative VP exclusively on human gameplay video. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.

[Concern 1] Positioning relative to exploration-based inverse dynamics learning and hindsight learning

Discussing these connections would help distinguish the contribution of video-guided data collection from the underlying learning principle.

We agree, and the revision separates the learning principle from our contributions. Learning inverse dynamics from hindsight-labeled, self-collected transitions is not new, and the related work now discusses BCO, GCSL, and AnyPos. We give particular attention to Grounding (Luo & Du, 2025), which other reviewers raised and which is the closest work to ours:

  • Video-guided data collection on long-horizon tasks. Grounding (Luo & Du, 2025) proposed video-guided exploration, but did not show that video models can guide exploration in long-horizon tasks. Our own experiments show that it fails to do so (Appendix D). To our knowledge, Mirror is the first method that leverages a large-scale video planner to solve long-horizon tasks, without action labels or rewards.
  • A V2A model conditioned on every future frame. In our ablation, replacing it with a single-subgoal policy lowers OGBench success from 0.91 to 0.64 (Appendix D, and Q2 below).

[Q1] Relation to AnyPos

Could you discuss Mirror relative to AnyPos, particularly when task-agnostic exploration provides sufficient coverage for action decoding?

The key difference is video-guided exploration. AnyPos (Tan et al., 2025) trains its IDM on random actions that uniformly cover a manipulator's workspace. This works where such actions cover the task, as in short-horizon manipulation. Long-horizon tasks such as Minecraft Diamond Challenge are different:

  • The action space is enormous. At each of 20 steps per second, the agent chooses among about half a million actions (≈5.2×105\approx 5.2 \times 10^5; Baker et al., 2022). A 20-minute episode thus admits more than 10137,00010^{137,000} action sequences, which can never be covered with random actions. Even human players trying to craft a diamond pickaxe obtain diamonds within 10 minutes in only 15% of episodes (Baker et al., 2022).
  • Random actions practically never break a single block. Breaking one requires holding the attack button for about 60 consecutive steps, which a policy pressing attack with probability 0.5 does with probability about 10−1810^{-18}. This is why DreamerV3 relies on accelerated mining and simplified crafting.
  • Later stages are never visited. The dark underground frames appear only after the agent has crafted a pickaxe and dug deep, so a randomly acting agent never collects data there.

[Q2] Relation to GCSL

Could you clarify the relationship between Mirror's training principle and hindsight-style method in GCSL?

The key difference is video-guided exploration. GCSL (Ghosh et al., 2021) gives its policy only a final goal, so nothing guides the agent on how to get there. This matters most when the goal is far away. Mirror's VP instead generates the intermediate plan, drawing on the prior of a large pretrained video model to show the next frames from the agent's current state toward the task. Apart from video-guided exploration, the learning principles are closely related: Mirror's training can be seen as GCSL's relabeling with the next chunk of frames as the goal. The remaining difference is that Mirror conditions on dense frame-to-frame transitions instead of a single distant goal, which also helps in our ablation (0.91 vs. 0.64 success on OGBench; Appendix D).

Dear Reviewer 5Scn,

As the discussion period draws to a close, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify any remaining points.

Thank you,
The Authors

I appreciate the response, which better clarifies the position of this paper in related work. I will maintain my score.

We thank the reviewer for the reply and for the positive assessment. We are glad that the response clarified the position of the paper in related work.

We would also like to point the reviewer to the new result in the revision. With a video planner trained exclusively on human gameplay video, Mirror now crafts an iron pickaxe without action labels or rewards, and it outperforms DreamerV3 and Dreamer 4 by a significant margin on the Minecraft Diamond Challenge. We believe this is a major breakthrough, and we would be grateful if the reviewer would take it into account. If any concern remains that keeps the reviewer from a higher score, we would be glad to address it.

Reviewer_4RwXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Mirror proposes a self-supervised approach for learning video-to-action (V2A) models without expert action labels or task rewards. Given an agent's observation history, a pretrained video planner predicts future frames, and the V2A model attempts to execute the corresponding actions with added exploration noise. The model then trains on the resulting observation-action transitions, gradually improving its ability to follow the generated plans. Replanning from the actual reached state is intended to keep exploration focused on difficult transitions until the agent learns to execute them.

The paper also develops a long-context video planner based on a 14B-parameter Cosmos3 model, incorporating block-causal attention, KV caching, sliding-window attention, and task-aware guidance. The method is evaluated on Minecraft BASALT, text-conditioned Minecraft tasks, and Minecraft Survival, as well as 24 tasks across DMC, OGBench, and Atari, using both generative and oracle video planners.

Critical strengths

1. A simple approach to learning action grounding without expert labels, with promising long-horizon results: The paper addresses an important limitation of video-based planning- translating generated visual trajectories into executable actions without relying on large, action-labeled datasets. Mirror provides a relatively simple approach based on self-collected transitions and demonstrates it in the full MineRL action space, with over 8,000 keyboard and mouse combinations.

The Minecraft results are particularly interesting. Mirror achieves non-zero success on MakeWaterfall without expert action labels or scripted behaviors and learns to craft a wooden pickaxe in Survival, where the VPT-BC baseline fails. These results suggest that stronger video planners could provide a useful source of supervision for learning increasingly complex behaviors through interaction.

2. Substantial video-planner engineering and evaluation across diverse environments: The adaptation of Cosmos3 into a long-context causal video planner is a valuable engineering contribution. The use of KV caching, sliding-window attention, variable-length predictions, and task-aware guidance allows the planner to operate over substantially longer horizons than its training clips.

I also appreciate the evaluation across Minecraft, DMC, OGBench, and Atari. The oracle-planner experiments are useful for separating limitations of the self-improvement algorithm from those of the video planner itself. The authors also plan to release the code and model weights, which would make these components useful for future work.

Critical weaknesses

1. The framing around learning without expert supervision overlooks important assumptions and makes some comparisons difficult to interpret: Although Mirror does not require expert action labels to train its V2A model, several experiments still rely heavily on expert supervision through the video planner.

  • Oracle VP: The oracle uses VPT's RL-trained policy to generate future observations in a cloned environment. This provides interactive expert guidance at the learner's current state. Consequently, the comparison showing an order-of-magnitude improvement in interaction efficiency over DreamerV3 is not directly comparable, since DreamerV3 learns using task rewards without access to an expert planner.
  • DMC, OGBench, and Atari: The generative video planners require DAgger recovery trajectories collected by querying an expert at learner-visited states. In contrast, BCO and LAIfO only receive fixed expert demonstrations.
  • Despite this additional supervision, Mirror with a generative planner underperforms LAIfO on DMC.

The action-label-free claim is technically valid for V2A training, but the broader supervision requirements should be stated more clearly. Results without DAgger recovery data, or comparisons where baselines receive equivalent expert information, would help establish how much improvement comes from Mirror itself.

2. Limited algorithmic novelty relative to prior work on self-improving video models and action grounding: The core learning mechanism is closely related to existing work. BCO [3] establishes the idea of learning inverse dynamics models from self-collected transitions. VideoAgent [1] already demonstrated how video planners/world models can self-improve through online environment interaction, refining the video generation model using newly collected experience without requiring expert action or reward labels and further improved with VLM feedback. While VideoAgent focuses on improving the video planner itself, subsequent work such as Grounding Video Models to Actions [2] addresses the action-grounding component through video-guided exploration and hindsight relabeling, learning executable policies without rewards or expert action labels. Mirror builds on closely related principles, but its distinction from these prior approaches could be better established.

Mirror's main algorithmic difference from Grounding Video Models to Actions[2] is learning frame-to-frame inverse dynamics rather than a goal-conditioned policy over longer horizons, along with frequent replanning from the agent's actual observations. However, the comparison with Grounding Video Models to Actions [2], does not fully isolate these differences. On OGBench, [2] follows a single generated plan during data collection, whereas Mirror replans every few steps. The resulting performance gap therefore reflects differences in both policy conditioning and replanning.

I think a more controlled comparison is necessary to support the claimed advantage of frame-to-frame V2A learning. More broadly, the paper would benefit from positioning its contribution relative to BCO, VideoAgent's action free reward free unsupervised objective for self-improving video planner and Grounding's reward-free action grounding, clarifying what is fundamentally new about Mirror's learning algorithm. Much of the contribution also appears to come from the video-planner architecture and its successful application to Minecraft, which deserves more emphasis in the main presentation.

3. The central mechanisms are not sufficiently ablated: The paper attributes Mirror's performance to several design choices, but provides limited quantitative evidence for their individual contributions.

Exploration noise is described as essential both for preventing policy collapse and for overcoming difficult milestones. Similarly, task-aware guidance is presented as important for maintaining goal persistence. However, the paper does not report ablations of exploration noise (including disabling it), noise magnitude or schedule, replanning frequency, task-aware guidance, or plan length.

The proposed milestone-crossing mechanism in Figure 2 is intuitive, but remains largely illustrative. These experiments would help establish whether the observed self-improvement comes from the proposed exploration mechanism and which components are actually necessary.

4. The Minecraft evaluation is limited in scale and uses protocols that are difficult to compare directly with prior work:

  • BASALT: Results are based on only 11 seeds per task, without confidence intervals. For instance, the reported 82% success corresponds to 9/11 episodes, compared with 7/11 for a 64% baseline. The evaluation also replaces BASALT's human judgments with author-defined objective criteria. Appendix B.1.2 indicates that baseline episodes were taken from Milani et al. (2023), but it is unclear who rescored them and whether the evaluation was blinded.
  • Text-conditioned tasks: The evaluation uses two author-designed environments with 10 episodes per instruction. Most baselines perform near zero, making it difficult to assess the strength of the evaluation. STEVE-1, an existing text-conditioned Minecraft agent included in Table 1, is notably absent from this comparison.
  • Minecraft Survival: The environment is modified with an enlarged inventory, cursor trails, and click indicators to accommodate limitations of the video planner. It is unclear whether VPT-BC was evaluated under the same conditions or how Mirror would perform without these modifications.

Given the importance of the Minecraft results to the paper's contribution, stronger evaluation protocols and clearer baseline comparisons would make the findings more convincing.

[1] Soni, Achint, et al. "VideoAgent: Self-Improving Video Generation." arXiv, 2024, https://arxiv.org/abs/2410.10076 [2] Luo, Yunhao, et al. "Grounding Video Models to Actions through Goal Conditioned Exploration." International Conference on Learning Representations, 2025, https://arxiv.org/abs/2411.07223 [3] Torabi, Faraz, et al. "Behavioral Cloning from Observation." Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, pp. 4950–4957, https://doi.org/10.24963/ijcai.2018/687 [4] Giammarino, Vittorio, et al. "Adversarial Imitation Learning from Visual Observations Using Latent Information." Transactions on Machine Learning Research, 2024, https://arxiv.org/abs/2309.17371

Questions for authors
  1. Could the authors clarify the supervision assumptions across the different settings, particularly access to an interactive expert and a clonable environment? How does generative-VP Mirror perform without DAgger recovery data, or when BCO and LAIfO receive equivalent expert information [3,4]?

  2. Could the authors compare Mirror and Grounding under matched replanning schedules, including Grounding with frequent replanning and Mirror without it? This would help isolate the claimed advantage of frame-to-frame inverse dynamics over goal-conditioned policy learning.

  3. Could the authors provide quantitative ablations of exploration noise (including disabling it), replanning frequency, and task-aware guidance? These seem central to the proposed self-improvement mechanism.

  4. For BASALT, were the baseline agents rerun or their recorded episodes rescored? Who applied the success criteria, and were evaluations blinded to the method? Could the authors report confidence intervals or results under the original human-judged BASALT protocol?

  5. Was VPT-BC evaluated with the same interface modifications used for Mirror in Minecraft Survival? How does Mirror perform without these modifications?

  6. What happens when the video planner repeatedly proposes transitions that the agent cannot execute? Can the agent remain stuck at a milestone indefinitely, and how frequently does this occur in practice?

  7. Why was STEVE-1 excluded from the text-conditioned Minecraft evaluation, despite being included in the BASALT comparison?

  8. Could the authors clarify how Mirror's self-improvement mechanism differs from existing work such as [1] which improves video planners through feedback and online interaction, [2] which already learns action grounding through reward-free, video-guided exploration and [3] Behavioral Cloning from Observation? Given the similarities in the underlying learning and exploration procedures, what are the main algorithmic contributions of Mirror beyond these approaches?

[1] Soni, Achint, et al. "VideoAgent: Self-Improving Video Generation." arXiv, 2024, https://arxiv.org/abs/2410.10076 [2] Luo, Yunhao, et al. "Grounding Video Models to Actions through Goal Conditioned Exploration." International Conference on Learning Representations, 2025, https://arxiv.org/abs/2411.07223 [3] Torabi, Faraz, et al. "Behavioral Cloning from Observation." Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, pp. 4950–4957, https://doi.org/10.24963/ijcai.2018/687 [4] Giammarino, Vittorio, et al. "Adversarial Imitation Learning from Visual Observations Using Latent Information." Transactions on Machine Learning Research, 2024, https://arxiv.org/abs/2309.17371

6 comments
AuthorsOpenReview ↗

Official Response by Authors (1/3)

We thank the reviewer for the detailed and constructive review. In brief:

  • New result: Mirror now sets a new state of the art on the Minecraft Diamond Challenge (Concern 1). We retrained the generative VP exclusively on human gameplay video in the revision. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.
  • Grounding proposed video-guided exploration, but did not show that video models can guide exploration in long-horizon tasks (Concern 2). Our own experiments show that it fails to do so. Mirror, in contrast, is the first method that leverages a large-scale video planner to solve long-horizon tasks, without action labels or rewards.
  • We ran the controlled comparison the reviewer requested. Mirror's frame-to-frame V2A model outperforms Grounding's goal-conditioned policy (Concerns 2 and 3).
  • We added the requested ablations of exploration noise, plan length, and replanning frequency (Concern 3). Frequent replanning matters, while Mirror is robust to the exact chunk size and noise level.

[Concern 1] Unclear supervision assumptions and hard-to-interpret comparisons

The framing around learning without expert supervision overlooks important assumptions and makes some comparisons difficult to interpret: Although Mirror does not require expert action labels to train its V2A model, several experiments still rely heavily on expert supervision through the video planner...

The Minecraft evaluation is limited in scale and uses protocols that are difficult to compare directly with prior work.

We agree that the initial submission was confusing on both points. Its generative Diamond VP was trained on a mixture of gameplay from humans and from VPT's RL agent, and its diamond result relied on an oracle VP built from an expert policy.

The revision resolves this directly. We retrained the generative VP exclusively on human gameplay video, so no expert policy is involved. We evaluate it on 128 episodes under the protocol of VPT and Dreamer 4. Without action labels or rewards, Mirror crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3, and still in 8.6% under DreamerV3's own protocol of 30-minute episodes that end at the first death (Appendix C.1.3). In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.

The revision also addresses the remaining points on the Minecraft evaluation: Mirror is additionally evaluated on 50 BASALT seeds with confidence intervals (Q4; Appendix C.1.2), the frames no longer show cursor trails or click indicators (Q5; Appendix B.1.7), and the text-conditioned experiment is framed as a proof of concept (Q7; Section 4.1.3).

Two settings still use expert policies, each for a specific reason.

  • The oracle VP is a deliberate control. It disentangles Mirror's action learning from the quality of the VP, asking how far Mirror can go without action labels or rewards when the VP is ideal. Mirror then obtains diamonds after about 80 hours of interaction, so the remaining bottleneck is the VP.
  • The control benchmarks use DAgger data for the VP. These benchmarks provide only synthetic expert data, which lacks diversity, so we augment the VP data with DAgger recoveries. Without DAgger, Mirror retains most of its performance on OGBench (0.80 vs. 0.91 success) and Atari (57% vs. 61% of expert), while its DMC return drops from 745 to 379.

The reviewer also noted that Mirror with a generative VP underperformed LAIfO on DMC. With the updated VPs, Mirror matches LAIfO on DMC (745 vs. 745 return) and outperforms it on OGBench (0.92 vs. 0.63 success) and Atari (62% vs. 9% of expert). More importantly, LAIfO learns by adversarially matching in-domain expert demonstrations online, so it cannot draw on a pretrained video model. Mirror is designed to turn the prior of large pretrained video models into actions. The control benchmarks use small VPs trained from scratch, so this advantage shows fully only in Minecraft, where the VP is pretrained at scale with rich human play data.

AuthorsOpenReview ↗

Official Response by Authors (2/3)

[Concern 2] Limited algorithmic novelty relative to prior work

Limited algorithmic novelty relative to prior work on self-improving video models and action grounding: ... the paper would benefit from positioning its contribution relative to BCO, VideoAgent's action free reward free unsupervised objective for self-improving video planner and Grounding's reward-free action grounding, clarifying what is fundamentally new about Mirror's learning algorithm.

The revised related work now states how Mirror differs from each of these works, including BCO, GCSL, VideoAgent, and Grounding. BCO also learns a V2A model from self-collected data, but it has no video model to guide its exploration. VideoAgent improves the video generator itself and extracts actions with optical flow, whereas Mirror keeps the VP fixed and learns its V2A model from scratch. The two are complementary. Grounding (Luo & Du, 2025) is the closest work to ours, and it first proposed the concept of video-guided exploration. We regret not crediting it for this, and the revision does.

However, Grounding did not show that video models can guide exploration in long-horizon tasks. Its own experiments use small video models trained from scratch on short tasks, and our experiments show that it cannot effectively leverage pretrained video models and fails on long horizons (Appendix D). Therefore, to our knowledge, Mirror is the first method that actually leverages a large-scale video planner to solve long-horizon tasks, without action labels or rewards.

Algorithmically, Mirror differs from Grounding in three ways. It uses a frame-to-frame V2A model in place of a goal-conditioned policy, simpler exploration and training, and replanning during training. In the controlled comparison the reviewer requested, each one improves performance with the paper's DAgger VPs. For example, replacing only the V2A model lowers OGBench success from 0.91 to 0.64 (Appendix D).

[Concern 3] Ablations of the central mechanisms

The central mechanisms are not sufficiently ablated: The paper attributes Mirror's performance to several design choices, but provides limited quantitative evidence for their individual contributions.

The revision ablates Mirror's components one at a time (Appendix D). In the chain from Mirror to Grounding, each change lowers performance.

  • V2A model. Replacing it with a goal-conditioned policy lowers OGBench success from 0.91 to 0.64 (DAgger VPs).
  • Exploration. Replacing Mirror's exploration noise with Grounding's alternation of random and video-guided episodes lowers it further, from 0.64 to 0.48 (DAgger VPs).
  • Replanning. Removing replanning during training lowers it from 0.27 to 0.05 (expert-only VPs).

The revision also adds the requested ablations of exploration noise, plan length, and replanning frequency (Q3; Appendix C.4). With the oracle VP on the four OGBench tasks, each variant changes a single factor of Mirror, and we report success after 500k environment steps.

  • Replanning frequency. Mirror replans after every chunk. Replanning less often eventually breaks it: with a new plan every two or three chunks, success stays at 0.90 and 0.87, but with a new plan every seven chunks it drops from 0.90 to 0.67, and on AntMaze from 0.93 to 0.18.
  • Plan length. Mirror needs action chunks but is robust to their exact length: chunks of 3 to 10 actions reach 0.88–0.93 success, while single-action chunks reach only 0.33.
  • Exploration noise. Disabling the noise lowers success from 0.90 to 0.86, while noise levels from 0.05 to 0.3 reach 0.89–0.95.

The proposed milestone-crossing mechanism in Figure 2 is intuitive, but remains largely illustrative.

While this mechanism is hard to prove directly, we added Figure 15 (Appendix C.1.5) as empirical evidence consistent with it. The figure shows how far each life of the agent got during online training on the Diamond Challenge: as training proceeds, the agent crosses progressively later milestones.

AuthorsOpenReview ↗

Official Response by Authors (3/3)

Answers to questions

  • Q1 (supervision assumptions). Please refer to Concern 1.
  • Q2 (matched comparison with Grounding). Please refer to Concerns 2 and 3.
  • Q3 (ablations). Please refer to Concern 3.
  • Q4 (BASALT). Please refer to Concern 1 and Appendix C.1.2, where we report Mirror on 50 seeds with confidence intervals. BASALT tasks are intentionally designed to be fuzzy and hard to verify, and we found automatic scoring with a VLM inaccurate. Success was therefore judged manually, using the criteria in Appendix B.1.2. The baselines' episodes are the recorded episodes of Milani et al. (2023), scored with the same criteria.
  • Q5 (interface modifications). Please refer to Concern 1 and Appendix B.1.7. In the revision, the frames no longer show cursor trails or click indicators. With the oracle VP, which does not go through the VAE, Mirror learns to mine diamonds without any of this post-processing.
  • Q6 (infeasible plans). In practice, infeasible plans were not a problem. When the agent fails to follow a plan, the VP replans from the agent's actual observation. Human gameplay is also full of recoveries, from which the VP learns to recover from mistakes.
  • Q7 (STEVE-1). The text-conditioned experiment is a proof of concept. Its purpose is to verify that language can steer the VP, and that a single task-agnostic V2A model can then carry out many tasks without ever knowing the task. It is not meant as a serious test of language understanding, nor to show that Mirror outperforms VLAs such as STEVE-1, which are trained specifically for Minecraft. The revision states this framing (Section 4.1.3).
  • Q8 (contributions over prior work). Please refer to Concern 2.

We hope this addresses the reviewer's concerns, and we would be grateful if the reviewer would consider updating the score.

Reviewer_rGFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  1. The authors propose Mirror, a self-supervised framework for learning video-to-action (V2A) models without expert action labels or task rewards.

  2. Mirror uses a pretrained video planner to guide exploration, while the V2A model learns from executed actions and achieved visual transitions.

  3. The authors adapt a 14B Cosmos3 video model with causal attention and KV caching for efficient long-horizon video planning.

  4. Mirror outperforms Grounding across DMC, OGBench, and Atari; achieves 82% FindCave and 36% MakeWaterfall success in Minecraft, and diamond acquisition with an oracle video planner.

Critical strengths
  1. Strong results on challenging Minecraft tasks without expert action labels. Mirror achieves 82% success on FindCave and 36% on MakeWaterfall using a V2A model trained entirely through self-collected interactions, outperforming several action-supervised baselines. Demonstrating this approach on complex, long-horizon Minecraft tasks is a meaningful practical contribution.

  2. Broad empirical evaluation with evidence for effective video-guided exploration. The paper evaluates 24 tasks across DMC, OGBench, and Atari, consistently outperforming the closely related Grounding (Luo & Du, 2025) method, with particularly large improvements on DMC and OGBench. Ablations comparing planner-guided and random exploration further demonstrate the benefit of using video plans to collect informative V2A training data.

Reference: Luo, Y., & Du, Y. (2025). Grounding Video Models to Actions through Goal Conditioned Exploration. International Conference on Learning Representations (ICLR 2025). https://arxiv.org/abs/2411.07223

Critical weaknesses

Limited algorithmic novelty and insufficient isolation of the proposed contributions. The core framework of video-guided exploration, stochastic action exploration, and self-supervised learning from achieved transitions is already established by Grounding (Luo & Du, 2025). The primary algorithmic distinction is conditioning an inverse dynamics model on a sequence of future frames rather than a single goal frame, which represents a relatively incremental extension. The causal KV-caching architecture also builds on established video-generation techniques. Although Mirror substantially outperforms Grounding, the comparisons use different video planners and different exploration procedures, making it difficult to attribute the gains specifically to the proposed action-model formulation. A controlled comparison using the same frozen video planner and exploration strategy is needed to establish the contribution.

Reference: Luo, Y., & Du, Y. (2025). Grounding Video Models to Actions through Goal Conditioned Exploration. ICLR 2025. https://arxiv.org/abs/2411.07223

Questions for authors

Isolating improvements over Grounding. Can the authors compare Mirror against Grounding using the same video planner, observation history, exploration strategy, and replanning frequency, particularly on Minecraft? This would help establish whether conditioning on multiple future frames, rather than differences in planning and exploration, is responsible for the reported improvements.

8 comments
AuthorsOpenReview ↗

Official Response by Authors (1/2)

We thank the reviewer for the careful review. In brief:

  • Grounding proposed video-guided exploration, but did not show that video models can guide exploration in long-horizon tasks (Concern 1). Our own experiments show that it fails to do so. Mirror, in contrast, is the first method that leverages a large-scale video planner to solve long-horizon tasks, without action labels or rewards.
  • We ran the controlled comparison the reviewer requested. With matched VPs, exploration, and replanning, Mirror's V2A model outperforms a goal-conditioned policy, including on Minecraft (Concerns 2 and 3).
  • New result: Mirror now sets a new state of the art on the Minecraft Diamond Challenge. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4. Unlike in the initial submission, these results are from our generative VP trained exclusively on human data. Building such a VP is a highly nontrivial contribution (Concern 2).

[Concern 1] Grounding has already established the framework

The core framework of video-guided exploration, stochastic action exploration, and self-supervised learning from achieved transitions is already established by Grounding (Luo & Du, 2025).

The reviewer is right that Grounding first proposed video-guided exploration. We regret not crediting it for this, and the revision does.

However, we respectfully disagree that it established the framework. The appeal of video-guided exploration is that large pretrained video models can guide exploration in long-horizon tasks that random exploration cannot solve, and Grounding's experiments test neither part (Appendix D).

  • Their motivation: Luo & Du (2025) motivate their method with "a large pretrained video model" and "challenging long-horizon tasks".
  • Their experiments: These, however, use small video models trained from scratch on very little data (15–20 demonstrations per task), and they leave finetuning large pretrained video models to future work. The tasks are short enough that a single plan, generated at the start of each episode, carries the agent from start to goal without replanning during training. Whether video-guided exploration scales to strong video planners and long horizons has therefore remained open.
  • What our experiments show: Grounding cannot effectively leverage pretrained video models. Planning a single subgoal image with frequent replanning consistently outperforms Grounding's multi-subgoal video planning strategy. Even when given an oracle planner, Grounding falls below LAIfO on DMC and OGBench, although LAIfO uses no video planner at all. Grounding also fails on long horizons: with the same oracle planner as Mirror, it does not acquire a single log in the Diamond Challenge.

For these reasons, Mirror is, to our knowledge, the first method that leverages a large-scale video planner to solve long-horizon tasks, without action labels or rewards. Please find more details in Concern 3 below and in Appendix D of our revision.

AuthorsOpenReview ↗

Official Response by Authors (2/2)

[Concern 2] Limited novelty: the VP and the V2A formulation

The primary algorithmic distinction is conditioning an inverse dynamics model on a sequence of future frames rather than a single goal frame, which represents a relatively incremental extension. The causal KV-caching architecture also builds on established video-generation techniques.

The VP is more than KV caching. We agree that KV caching is established. However, turning a 14B bidirectional video model into a long-context, closed-loop planner also required block-causal adaptation, variable-length plans, sliding-window attention over a fixed-size cache, and task-aware guidance. Reviewer 4RwX also describes this adaptation as "a valuable engineering contribution". To show the effectiveness of our VP, we retrained the generative VP exclusively on human gameplay video in the revision. It now sets a new state of the art in Dreamer 4's offline setting. It obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4. Without action labels or rewards (online setting), Mirror crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3.

The V2A formulation matters. With the paper's DAgger VPs, replacing only Mirror's V2A model with a goal-conditioned policy that reaches a single subgoal lowers performance on every benchmark, e.g., from 0.91 to 0.64 success on OGBench. On the Diamond Challenge with the oracle VP, the same change lowers the crafting-table rate from 76% to 2%. Mirror also differs from Grounding in its exploration, training, and replanning, and each of these further improves performance (Concern 3; Appendix D).

[Concern 3] A controlled comparison with the same VP, particularly on Minecraft

Can the authors compare Mirror against Grounding using the same video planner, observation history, exploration strategy, and replanning frequency, particularly on Minecraft?

We ran this comparison in the revision. Starting from Mirror, we remove its three contributions one at a time, with VPs trained on the same data: (a) the V2A model, (b, c) its exploration and training, and (d) replanning during training (Table 7). Please find the details in Appendix D.

  • Changing only the policy lowers performance. Variant (a) replaces the V2A model with a single-subgoal policy and keeps Mirror's observation history, exploration noise and replanning frequency, as the reviewer asks. Its VP is trained on the same data as Mirror's but predicts a single subgoal one action chunk ahead. With DAgger VPs, Mirror outperforms it on every benchmark (0.91 vs. 0.64 success on OGBench, 745 vs. 546 return on DMC, 61% vs. 54% of expert on Atari). With expert-only VPs the same holds except on DMC (379 vs. 460).
  • Full Grounding is far behind. With all three contributions removed, on the same expert data, Grounding reaches 0.05 success on OGBench against Mirror's 0.80, 179 return on DMC against 379, and −4% of expert on Atari against 57%. Even the version in our paper, which adds Mirror's replanning on DMC and Atari, stays far behind with DAgger VPs (293 vs. 745 return on DMC, 36% vs. 61% of expert on Atari).
  • On Minecraft, Grounding does not reach the first tree. Both methods use the same oracle VP, so they follow identical future frames. Grounding acquires no wood logs even after 1M environment steps, while Mirror acquires them in 100% of episodes. Videos are on our anonymous webpage (https://selfimprovingv2a.github.io/). We did not repeat this with our generative Minecraft VP, because that VP is one of our contributions and pairing it with Grounding would be an ablation of Mirror.

We hope this addresses the reviewer's concerns, and we would be grateful if the reviewer would consider updating the score.

Dear Reviewer rGFQ,

As the discussion period draws to a close, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify any remaining points.

Thank you,
The Authors

Comments on the paper

1 comment
Area_Chair_ZVauOpenReview ↗

Discussion with the authors

Dear reviewers,

the authors have provided responses for your reviews. Can you please let them know if your comments have been addressed?

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.