- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Mirror Learning, a method for learning robot policies from third-person videos. The main idea is to use a video diffusion model to convert third-person observations into first-person views, and then use an inverse dynamics model to predict the corresponding actions. The generated data can then be used to train a policy through behavior cloning. Experiments in CARLA and Minecraft show that this approach can improve policy performance, even when using only the generated demonstrations.
Critical strengths
- Interesting and practical idea. Learning from third-person videos is an important problem, as collecting first-person demonstrations can be expensive. The proposed approach provides a reasonable way to turn these videos into useful training data.
- Good experimental results. The authors evaluate the method in both CARLA and Minecraft. The results show that the generated data can improve behavior cloning, and the additional closed-loop experiments provide further support for the approach.
Critical weaknesses
- Still requires considerable supervision. Training the video model requires paired first-person and third-person videos, and the pipeline also relies on segmentation and tracking. This makes the method less scalable than the paper seems to suggest.
- Limited real-world and baseline evaluation. The real-world results are mostly qualitative, and the closed-loop evaluation is relatively small. It would also be helpful to compare against stronger viewpoint-transfer baselines.
Questions for authors
- How well does the method work when the segmentation or tracking is imperfect? Can the pipeline be applied to real-world videos without manually selecting demonstrators?
- Have the authors considered comparing against geometry-based viewpoint transformation methods? This would help clarify how much the video diffusion model contributes.
Thank you for your thoughtful review. We’re glad you found the problem we studied is important and our approach interesting and practical.
W1 Still requires considerable supervision.
We agree that training the MVM requires paired views, as noted in our conclusion. This step needs no action labels from either perspective, and once trained, the MVM and inverse dynamics model generate demonstrations from new videos without paired views or action labels. In our driving experiments, the pairs come from simulation, MVM is able to generate realistic data zero-shot. (Figs. 3, 13, 14).
In Minecraft and on robotaxi dataset, we use pretrained SAM 3 out of the box to segment and track agents. Demonstrators are chosen by fixed filters on visibility and track length (App. A.2.2 and A.3). We have reworded "manually select" in Sec. 3. In the robotaxi study, manual selection was used only to find paired data for comparison with ground truth (App. A.2.2). At inference time, the MVM needs only the learner’s video and a demonstrator mask, and the demonstrator can be selected by visibility and track-length filters.
W2 Broader evaluations
We agree that broader real-world and closed-loop evaluations would strengthen the paper. We used the limited robotaxi footage publicly available to us to assess zero-shot transfer of the MVM. We think that access to paired real-world driving data would likely enable further gains which we leave for future work. We also added a comparison with a privileged geometry-based method, described under Question 2.
Q1 "How well does the method work when the segmentation or tracking is imperfect?" and "manually selecting demonstrators"
For the Minecraft and robotaxi dataset, masks come from SAM 3, so they contain segmentation and tracking errors. In highly dynamic environments, such as our real human-played multi-agent Minecraft dataset, 40% of clips contain at least one frame in which the demonstrator is not visible, and in 11% of clips the demonstrator is absent from at least one-sixth (16.7%) of the frames (App. A.3). The MVM uses context from the remaining frames to generate a temporally coherent demonstrator perspective video.
We also tested this on 64 CARLA validation clips by randomly removing the demonstrator mask from 20% of total frames, a condition that the MVM trained on Carla dataset never saw during training (App. A.12). The IDM error on the resulting mirror videos changes by 0.019 m, comparable to resampling the generator seed (0.013 m).
Please see W1 for working with real-world videos.
Q2 Geometry-based viewpoint transformation
Thank you for suggesting this comparison. Geometry-based methods require depth and camera poses, which our method does not use, so they solve a substantially easier problem. To show what geometry can achieve at best, we evaluated multi-frame reprojection with ground-truth depth and camera poses (for both learners and demonstrators) (App. A.11). Even then, it can only fill regions the learner observed (85% of the demonstrator's view on average using surrounding frames), and it assumes a static scene, so moving vehicles are misplaced.
We compare our MVM and this privileged reprojection alternative approach against the demonstrator’s recorded first-person view on 64 six-second clips from the validation set.
Using only the learner’s RGB video and the demonstrator mask, our MVM scores better than the reprojection alternative on all three video metrics. The video model generates the regions the learner never sees, and it does so without ground truth depth or camera poses.