acceptodds
Under review as a conference paper at ICLR 2027

Mirror Learning: Learning Egocentric Policies from Third-Person Observations

Abstract

We investigate imitation learning through the lens of third-person observation and propose a method for mirror learning: acquiring actionable policies from passive observation. While behavior cloning (BC) excels under dense, well-aligned first-person data, it fundamentally fails to leverage the rich observational signals arising from third-person demonstrations that humans and animals routinely exploit. We introduce a method that composes (i) a learned perspective transformation that places learners in demonstrators' shoes using a fine-tuned video diffusion model and (ii) an inverse dynamics model that infers action trajectories in the learners' control space. Once trained, this enables the synthesis of mirror data, pseudo first-person expert data generated from third-person observations of demonstrator behavior. Empirically, we show that mirror data alone can train effective policies, and that augmenting first-person BC training with mirror data further improves downstream policy performance. Our results suggest that modern generative world models implicitly encode sufficient structure to enable a scalable alternative to teleoperation-heavy data collection.

Primary area:
applications to robotics, autonomy, planning
Keywords:
imitation learning, learning from observation, video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ye4iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mirror Learning, a method for learning robot policies from third-person videos. The main idea is to use a video diffusion model to convert third-person observations into first-person views, and then use an inverse dynamics model to predict the corresponding actions. The generated data can then be used to train a policy through behavior cloning. Experiments in CARLA and Minecraft show that this approach can improve policy performance, even when using only the generated demonstrations.

Critical strengths
  • Interesting and practical idea. Learning from third-person videos is an important problem, as collecting first-person demonstrations can be expensive. The proposed approach provides a reasonable way to turn these videos into useful training data.
  • Good experimental results. The authors evaluate the method in both CARLA and Minecraft. The results show that the generated data can improve behavior cloning, and the additional closed-loop experiments provide further support for the approach.
Critical weaknesses
  • Still requires considerable supervision. Training the video model requires paired first-person and third-person videos, and the pipeline also relies on segmentation and tracking. This makes the method less scalable than the paper seems to suggest.
  • Limited real-world and baseline evaluation. The real-world results are mostly qualitative, and the closed-loop evaluation is relatively small. It would also be helpful to compare against stronger viewpoint-transfer baselines.
Questions for authors
  • How well does the method work when the segmentation or tracking is imperfect? Can the pipeline be applied to real-world videos without manually selecting demonstrators?
  • Have the authors considered comparing against geometry-based viewpoint transformation methods? This would help clarify how much the video diffusion model contributes.
1 comment

Thank you for your thoughtful review. We’re glad you found the problem we studied is important and our approach interesting and practical.

W1 Still requires considerable supervision.

We agree that training the MVM requires paired views, as noted in our conclusion. This step needs no action labels from either perspective, and once trained, the MVM and inverse dynamics model generate demonstrations from new videos without paired views or action labels. In our driving experiments, the pairs come from simulation, MVM is able to generate realistic data zero-shot. (Figs. 3, 13, 14).

In Minecraft and on robotaxi dataset, we use pretrained SAM 3 out of the box to segment and track agents. Demonstrators are chosen by fixed filters on visibility and track length (App. A.2.2 and A.3). We have reworded "manually select" in Sec. 3. In the robotaxi study, manual selection was used only to find paired data for comparison with ground truth (App. A.2.2). At inference time, the MVM needs only the learner’s video and a demonstrator mask, and the demonstrator can be selected by visibility and track-length filters.

W2 Broader evaluations

We agree that broader real-world and closed-loop evaluations would strengthen the paper. We used the limited robotaxi footage publicly available to us to assess zero-shot transfer of the MVM. We think that access to paired real-world driving data would likely enable further gains which we leave for future work. We also added a comparison with a privileged geometry-based method, described under Question 2.

Q1 "How well does the method work when the segmentation or tracking is imperfect?" and "manually selecting demonstrators"

For the Minecraft and robotaxi dataset, masks come from SAM 3, so they contain segmentation and tracking errors. In highly dynamic environments, such as our real human-played multi-agent Minecraft dataset, 40% of clips contain at least one frame in which the demonstrator is not visible, and in 11% of clips the demonstrator is absent from at least one-sixth (16.7%) of the frames (App. A.3). The MVM uses context from the remaining frames to generate a temporally coherent demonstrator perspective video.

We also tested this on 64 CARLA validation clips by randomly removing the demonstrator mask from 20% of total frames, a condition that the MVM trained on Carla dataset never saw during training (App. A.12). The IDM error on the resulting mirror videos changes by 0.019 m, comparable to resampling the generator seed (0.013 m).

Please see W1 for working with real-world videos.

Q2 Geometry-based viewpoint transformation

Thank you for suggesting this comparison. Geometry-based methods require depth and camera poses, which our method does not use, so they solve a substantially easier problem. To show what geometry can achieve at best, we evaluated multi-frame reprojection with ground-truth depth and camera poses (for both learners and demonstrators) (App. A.11). Even then, it can only fill regions the learner observed (85% of the demonstrator's view on average using surrounding frames), and it assumes a static scene, so moving vehicles are misplaced.

We compare our MVM and this privileged reprojection alternative approach against the demonstrator’s recorded first-person view on 64 six-second clips from the validation set.

MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
MVM20.310.5850.208
Privileged Reprojection16.140.5550.435

Using only the learner’s RGB video and the demonstrator mask, our MVM scores better than the reprojection alternative on all three video metrics. The video model generates the regions the learner never sees, and it does so without ground truth depth or camera poses.

Reviewer_BVtHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a methodology for training RL policies via behavior cloning on a dataset made up of, or augmented with, mirror data. Mirror data takes advantage of implicit third-person demonstrations in the environment and can be extracted by first learning a perspective transformation between the learner and the demonstrator to obtain new observations and then using an Inverse Dynamics Model to fill in the missing action labels. The authors test their method on an autonomous driving task and on Minecraft and show that augmenting available data with mirror data improves policy learning and that it is even possible to train a policy on mirror data alone.

Critical strengths
  1. The paper is well written, structured and clear throughout. The motivation is well presented and problem setting and methodology are rigorously defined. The experimental scenarios are well well justified in the context of the research questions.

  2. The paper addresses a relevant problem which is that of taking advantage of implicit third-person demonstration data without the need for labelling.

  3. The authors demonstrate their method in two very distinct scenarios (Minecraft and autonomous driving).

  4. Results show that augmenting available data with mirror data improves policy learning and that mirror data alone can yield good policies, even though it may require larger amounts of data than GT first-person data.

Critical weaknesses
  1. There is some hallucination in Figure 3, with a car that doesn’t exist in the GT being placed in the mirrored image and pedestrians crossing the street disappearing. While this is in part justified by the lack of pedestrians in the CARLA dataset, the MVM still needs to inpaint occlusions, fundamentally having to guess what lies in those regions. Could contaminating the data with such biases result in unsafe behaviour? I think it would be important to improve the discussion on such potential dangers and limitations, possibly in a limitations section.
Questions for authors
  1. Did you consider alternatives to the video flow matching model? For example by learning a shared latent space like in Time-Contrastive Networks?
2 comments

Thank you for your thoughtful review. We're glad you found our motivation clear and the experiments well suited to the research questions.

W1 Hallucinations and limitations

In Figure 3, the learner cannot see the road immediately ahead of the demonstrator. The MVM learns where surrounding agents are likely to be given what the learner can see, so a car in that occluded region is a plausible sample from its learned conditional distribution. We agree that some pedestrians disappear in the mirrored video. This is a zero-shot example, where the MVM was fine-tuned only on CARLA data that contained no pedestrians. We included it because the MVM still renders some pedestrians, which suggests that it can learn the viewpoint transformation task from synthetic data while retaining some visual knowledge from the pretrained model. We expect fine-tuning on a small amount of real paired data to reduce such errors (Sec. 4.3).

Thank you for raising the safety concern. We agree that the MVM can generate scene content that differs from what actually happened, which can introduce biases into the mirror data. A driving policy trained on such data could learn unsafe behavior. We discuss this risk in the revised limitations section (Sec. 7).

Q1 For example by learning a shared latent space like in Time-Contrastive Networks?

Thank you for the suggestion. TCN (Sermanet et al., 2018) uses contrastive learning to align frames from different views of the same event, and the distance between their embeddings can serve as an imitation reward for RL. Our setting has no reward function or environment interaction during training (Sec. 2). In addition, several agents can appear in the same video, and comparing frame-wise embeddings does not give us a way to specify which agent to imitate. Our demonstrator mask makes that choice explicit. Generating RGB video also means that any end-to-end visual policy with the same observation and action format can train on the resulting pairs, without adapting to a new latent representation. We think self-supervised approaches like TCN remain a promising direction, and we have added a discussion to the revised manuscript (Sec. 5).

I thank the authors for their clarifications and for the additions to the manuscript. After consideration, I believe my initial scores continue to be fair.

Reviewer_T5dZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interesting idea to augment the training data for vision-based end-to-end driving policies by finetuning Cosmos to generate imaginary data of nearby vehicles. Its trajectories are then reconstructed using an inverse dynamics model.

Critical strengths

The paper leverages world model in an interesting way. It can potentially augment the value of mined interactive clips by generating trajectories of potentially high learning signals.

Critical weaknesses

My main concern is the paper does not study several important aspects of the presented approach:

  • How to ensure the generated clips does not hallucinate? The critical value of the approach, in my opinion, relies on being faithfully reconstruct novel / third-person view of some interesting interactive scenarios. Hence the generated data must reflect the ground truth interaction, otherwise the scenarios could lose its significant semantic meaning
  • Similar to above, measure the distance to ground-truth of all actors in the scene to measure how faithful the video represent the scenario.
  • How far of objects in the original clip from whose view can the method generate, and for how long?
Questions for authors

Please see the weakness section above.

3 comments

We thank Reviewer T5dZ for recognizing the potential of our approach and asking whether mirror videos preserve the original interactions. Our existing experiments show how accurately we recover the demonstrator’s motion. Our new localization study shows that mirror videos closely preserve the positions of vehicles visible to the learner.

Q1-2 "How to ensure the generated clips does not hallucinate?" and "measure the distance to ground-truth of all actors in the scene"

We agree that our method requires realistic generation of demonstrator observations. Occlusion and differences in viewpoint leave some agents' positions uncertain, so the learner's video does not uniquely determine the demonstrator’s view. We instead train the MVM to model the conditional distribution of the demonstrator's first-person video given the learner's first-person video and the demonstrator mask (Sec. 3). Samples from this distribution may differ from the ground truth and still be plausible given the limited conditional information. Our goal is to generate samples realistic enough to supervise a downstream policy.

For qualitative evidence, see Figures 1, 3, 4, and 10-14. Quantitatively, we refer to Table 8, where trajectories recovered by the IDM from mirror video have a displacement error 0.22m higher than those recovered from ground truth video.

As requested, we compare vehicle positions in generated and recorded videos on 192 CARLA clips, with four samples per clip (Sec. 4.3, App.A.10). Because 3D positions are challenging to estimate accurately from monocular video, we compare 2D boxes from the same pretrained detector, paired by Hungarian matching. We report the fraction of recorded boxes matched and the mean intersection-over-union (IoU) of matches.

For vehicles visible to both the learner and the demonstrator, 92% of bounding boxes detected in the recorded demonstrator video are matched to boxes detected in the generated video at IoU ≥\geq 0.5, with a mean IoU of 0.80 among these matches (97% are matched at IoU ≥\geq 0.1). For reference, a 4-pixel offset between two 36x36 pixel boxes gives an IoU of 0.80, a lateral shift of 0.11 m at 10 m and 0.33 m at 30 m. For hidden vehicles, 86% of recorded demonstrator boxes in the 10-20 m distance bin (to the learner) match a generated box at IoU ≥\geq 0.5, with a mean IoU of 0.78 among matches. The match rate decreases with distance from the learner (Table 2), as distant hidden vehicles are the least constrained by the learner's observations.

Our central question is whether supervision recovered from third-person demonstrations is sufficiently accurate to train and improve an egocentric behavior cloning policy. Policies trained on unfiltered mirror data improve both driving and Minecraft experiments. (Table 1 and Figure 5b; e.g., held-out-town minADE5 improves from 0.85 to 0.76 when mirror data are added), even when observations may differ from the ground truth. We agree that exact reproduction may matter more when mining highly interactive scenarios, a promising direction that we illustrate with a cut-in example (Fig.22). Our revised limitations section discusses hallucination, and future work could use rejection sampling to favor generations that better preserve the visible interaction.

Q3 How far of objects in the original clip from whose view can the method generate, and for how long?

The model learns from the distances and viewpoints represented in its training data. In CARLA, our method generates the view of a demonstrator that is within 24 m of the learner and visible to its front-facing camera, for 6 seconds (App. A.2.1). We chose the 6-second window to manage compute. A promising direction for future work is to use additional cameras on the learner to help it follow the demonstrator through longer interactions.

Dear Reviewer T5dZ, as the discussion period is ending soon, we would appreciate your feedback on whether our responses and new results help address your concerns. Thank you for your time and consideration.

Dear reviewer T5dZ,

As we approach the end of the discussion period, we'd be very grateful for any thoughts you'd like to share on our response and new results. We think the additional results we have included have addressed many of the weaknesses raised in your initial review, but we would welcome your feedback.

On Q1 and Q2, our new localization study shows that, for vehicles visible to the learner, 92% of boxes detected in the demonstrator’s recording matched boxes in the generated view at IoU ≥\geq 0.5, with a mean IoU of 0.80 among matches (Sec. 4.3, Table 2). Figure 22 additionally shows a recorded and generated cut-in side by side alongside our existing qualitative results.

On Q3, we clarified that our CARLA setup selects demonstrators within 24 m of the learner and generates their first-person view over a 6-second window (App. A.2.1).

Thank you again for your ongoing participation in the review process, we hope to hear from you soon.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their thoughtful feedback. We appreciate that the reviewers found the problem important (ye4i), the idea interesting (T5dZ, ye4i) and practical (ye4i), the paper clear (BVtH, ye4i), and the experiments in driving and Minecraft well justified (BVtH) with good results (ye4i). Our central question is whether supervision recovered from third-person demonstrations is sufficiently accurate to train and improve an egocentric behavior cloning policy. Below we summarize the main questions, new results and changes to the revisions.

 

(T5dZ Q1) Is the interaction preserved?

The learner’s video does not fully determine the demonstrator’s view, so we train the MVM to model a conditional distribution, and samples may differ from the ground truth in regions hidden from the learner. In our new localization study, for vehicles visible to the learner, 92% of bounding boxes detected in the recorded demonstrator video are matched to boxes detected in the generated video at IoU ≥\geq 0.5, with a mean IoU of 0.80 among these matches (97% are matched at IoU ≥\geq 0.1). For reference, a 4-pixel offset between two 36x36 pixel boxes gives an IoU of 0.80 and corresponds to a lateral shift of 0.11 m for a vehicle 10 m away and 0.33 m for one 30 m away under our camera configuration.

In addition, we refer to our existing Table 8, where trajectories recovered by the IDM from mirror video have a displacement error 0.22m higher than those recovered from ground truth video.


(BVtH, Q1) Limitations and safety

We added a limitations section. It discusses how content generated in regions hidden from the learner can differ from what actually happened, and the risk this poses for driving policies trained on mirror data.


(ye4i, Q1) Imperfect demonstrator masks

Removing the demonstrator mask from 20% of total frames, a condition the MVM trained on Carla dataset never saw during training, changes the IDM error on mirror videos by 0.019 m, comparable to resampling the generator seed (0.013 m).

(ye4i, Q2) What does video generation contribute?

We compared our MVM with a geometry-based reprojection alternative. Reprojection solves a substantially easier problem, because it is given ground-truth depth and the camera poses of both agents, which our method does not use. Even so, it cannot show what the learner never saw, and it assumes a static scene. The video model generates the unseen regions and the moving vehicles from RGB video and a mask alone, and it scores better than the reprojection alternative on all three video metrics (PSNR 20.31 vs 16.14).


Main additions in our revision

AdditionReviewerLocation
Vehicle localization studyT5dZSec. 4.3, App. A.10
Limitations section covering hallucination and safetyBVtH, T5dZSec. 7
Geometry-based reprojection baselineye4iApp. A.11
Mask robustness experimentye4iApp. A.12
Discussion of TCNBVtHSec. 5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.