- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how to learn an inverse dynamics model (called a V2A model) from an expert visual planner through self-collected experience. Rather than relying on random exploration, the visual planner is utilized to keep replanning meaningful rollouts to bring the agent back to the task-specific stuck points until the inverse dynamics model outputs a correct action to overcome it. The inverse dynamics model can also structurally learn from arbitrary quality experience. The authors demonstrate their method for learning IDMs on a variety of simulated environments such as Minecraft, DMC, OGBench, and Atari.
Critical strengths
The strength of this paper is its clarity of writing, and its straightforward approach. Furthermore, its findings support the scaling of video modeling efforts, showing how they can facilitate robotic decision-making through visual planning.
Critical weaknesses
One weakness of this work is its assumed access to an expert video generative model. This should not be assumed to be easy to get, for arbitrary practical robotic settings. Different robot embodiments may have different visual viewpoints, and adapting expertness into the video model correspondingly may turn into an expensive and adhoc endeavour.
Another weakness of this work is its contextualization with prior work, which has investigated self-improvement for visual planners. In particular, SILVR [1] performs self-improvement on both the video model as well as the IDM, whereas this proposed work only self-improves the IDM. SILVR also found sample efficiency benefits in self-improvement through visual planning, overlapping with the findings and takeaways of this work; but unlike this work, SILVR does not expect access to an expert video generative model (simply one that has adapted some level of environment visuals and dynamics [2]). This seems a more general setting - how quickly can a complete visual planner adapt to a new task, for which we do not assume expert behavior a priori, versus assuming a video model already encodes successful behavior within it. Similar prior work also look into self-improvement for WAMs, which can be thought of a singular model that jointly predicts video and actions (rather than the hierarchical approach of a video model and IDM) - it would be good to highlight efforts to update WAMs with respect to online rollout experience as well [3].
Another weakness is its evaluation suite. Particularly for synthetic environments, an argument could be made that interaction data is cheap and easy to collect - particularly in the age of agents that can spin up virtual environments and perform arbitrary interactions at scale. IDMs (or V2A models as the authors call them) can be trained on arbitrary quality interaction data, so such scaled generation data gathered from synthetic environments should result in a strong IDM up front - studying how they can be self-improved from scratch seems rather contrived for synthetic environments. Rather, it is learning how to achieve an expert-level video model that is interesting for synthetic environments, as this cannot be obtained from agents randomly interacting in synthetic environments at scale - but this work assumes access to an expert video model a priori. Thus, by only using synthetic environments, the evaluation setup of this work seems contrived. On the other hand, a convincing argument could be made for evaluating Mirror on real-world environments. In real-world settings, potentially different robots have different embodiments and action spaces - and a high-quality IDM may not necessarily be assumed a priori. A real-world experiment which takes an expert-level video model for a particular task, and reuses it for a new embodiment and action space (but, for example, keeps the camera angle the same for transferability), would be interesting and compelling evaluation, as agents cannot (currently) generate scaled interaction data in the real world for arbitrary embodiments. Thus, this reviewer believes this work would benefit from real-world experiments to be convincing.
Furthermore, as a small additional note on the synthetic experiment suite: as visual planners are expensive during inference, their benefit mostly lies in their high-level planning ability. However, apart from Minecraft, the other benchmark tasks such as OGBench do not particularly seem to benefit strongly from high-level visual planning.
[1] Luo et al., Self-Improving Loops for Visual Robotic Planning, ICLR 2026.
[2] Luo et al., Solving New Tasks by Adapting Internet Video Knowledge, ICLR 2025.
[3] Kim et al., Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning, arXiv 2026.
Questions for authors
In terms of replanning capabilities, how can the video model consistently guide the agent towards milestones it has failed on? Does this not require an even stronger video model than one that only encodes expert behavior - but a video model that can even perform recovery from failure or random states? If so, is this not a prohibitively expensive requirement to make of a video model for an arbitrary environment of interest (e.g. perfect expert recovery)?
8 comments
Official Response by Authors (2/2)
[Concern 3] IDMs are easy to train from arbitrary-quality interaction in synthetic environments
IDMs (or V2A models as the authors call them) can be trained on arbitrary quality interaction data, so such scaled generation data gathered from synthetic environments should result in a strong IDM up front.
In long-horizon tasks, random interactions never achieve sufficient coverage. In short-horizon tasks, random interactions can provide sufficient coverage to train an IDM. In long-horizon tasks, however, they cannot.
- The action space is enormous. At each of 20 steps per second, the agent chooses among about half a million actions (; Baker et al., 2022). A 20-minute episode thus admits more than action sequences, which can never be covered with random actions. Even human players trying to craft a diamond pickaxe obtain diamonds within 10 minutes in only 15% of episodes (Baker et al., 2022).
- Random actions practically never break a single block. In Minecraft, breaking a block requires holding the attack button for about 60 consecutive steps, which a policy pressing attack with probability 0.5 does with probability about . This is why DreamerV3 relies on accelerated mining and simplified crafting, whereas Mirror learns mining from the VP's guidance alone.
- Our experiments confirm this. Random-V2A, which is Mirror's V2A model trained only on random interaction, reaches 227 return on DMC against 788 for Mirror, and 0.38 success on OGBench against 0.93, both with the oracle VP. In text-conditioned Minecraft, a V2A model trained on random walks never chops a tree, while Mirror succeeds in 100% of episodes.
[Concern 4] Real-world experiments
Thus, this reviewer believes this work would benefit from real-world experiments to be convincing.
We focus on simulation because we target long horizons. We agree that real-world experiments would be valuable. However, running real robots on long-horizon tasks, where video-guided exploration matters most, is currently difficult at academic scale. Our self-improvement loop runs for 140 hours of interaction. Data is also a problem: we know of no public real-robot video data that has the horizon and diversity of human Minecraft gameplay, with episodes of up to 20 minutes that include many natural mistakes and recovery behaviors.
We also highlight that for short-horizon real-world tabletop settings, VERA (Li et al., 2026) has already shown that a frozen VP paired with an IDM trained on self-play data works well on real robots. This holds even for a 16-DoF Allegro hand, a contact-rich dexterous setting that usually requires large-scale RL in simulation. Thus, short horizons are already within reach, and long horizons remain the open problem. The difficulty comes from the horizon, not from whether the environment is synthetic or real. This is why we focus on long horizons.
[Concern 5] Benefit of visual planning on OGBench
Apart from Minecraft, the other benchmark tasks such as OGBench do not particularly seem to benefit strongly from high-level visual planning.
We agree that the control benchmarks test action grounding, not planning. This is by design. Mirror's primary purpose is to turn the prior of large pretrained video models into actions, and the control benchmarks isolate this step. There, given a capable VP, we test whether Mirror grounds it into actions better than standard learning-from-observation baselines such as LAIfO and VIPER. We demonstrate the benefit of long-horizon visual planning in Minecraft.
We also note the limits of these baselines. LAIfO learns by adversarially matching in-domain expert demonstrations online. It therefore cannot draw on a pretrained video model, and in our experience its training is unstable and sensitive to hyperparameters. VIPER does draw on a pretrained video model, but it is not entirely free from adversarial dynamics either, due to its exploration term. It is also prone to reward hacking, as video likelihoods become unreliable in high dimensions.
We hope this addresses the reviewer's concerns, and we would be grateful if the reviewer would consider updating the score.
Dear Reviewer Kev5,
As the discussion period draws to a close, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify any remaining points.
Thank you,
The Authors
Official Response by Authors (1/2)
We thank the reviewer for the feedback and for the pointers to related work. In brief:
[Concern 1] Contextualization with SILVR and prior work
The revised related work now discusses all three works the reviewer points to, along with other prior work. We added SILVR (Luo et al., 2026), the video adaptation work of Luo et al. (2025), and Cosmos Policy (Kim et al., 2026), as well as VideoAgent, BCO, GCSL, and AnyPos.
We also discuss combining Mirror with SILVR as future work. In particular, we clarify how Mirror relates to SILVR:
[Concern 2] Assumed access to an expert video model
Our revised VP involves no expert policy. In the revision, we retrained the generative VP exclusively on human gameplay video, without actions or rewards. With it, Mirror sets a new state of the art on the hour-long Diamond Challenge: it crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3, and obtains diamonds in 5.5% of episodes in Dreamer 4's offline setting, against 0.7% for Dreamer 4. A VP trained only on human video thus goes surprisingly far. We agree that obtaining such a VP is not easy, which is why our VP architecture and training recipe are important contributions.
Capable VPs are becoming available. At short horizons in the real world, LVP (Chen et al., 2025) and VERA (Li et al., 2026) show that capable VPs can be trained from Internet-scale video and executed on real robots. Our results suggest that the same is possible at much longer horizons.
Human gameplay already teaches the VP to recover. No additional data or supervision is needed. Human gameplay is full of mistakes and recoveries, so a VP trained on it learns to recover from failure states. The VP also replans from the agent's actual observation, so it always plans from wherever the agent has ended up. Videos of replanning and recovery are on our anonymous webpage (https://selfimprovingv2a.github.io/).